ChkWebAuditBot
ChkWebAuditBot/1.0 (+https://chkweb.com/bot)
If you found this page in your access log, that is exactly why it exists. ChkWebAuditBot crawls public pages to produce an SEO audit, the way Googlebot, Screaming Frog and Ahrefs do. Somebody asked for an audit of your site — often you, sometimes an agency or a developer working on it.
How to stop it
Add this to the robots.txt at the root of your site. We read robots.txt before anything else on every run, and a rule that names us is obeyed even when a wildcard group would allow the crawl.
User-agent: ChkWebAuditBot
Disallow: /It takes effect on the next run — there is nothing to wait for and nobody to email. To keep us out of one section instead, disallow that path rather than the whole site.
How to let it in
If your firewall, bot protection or host turns away crawlers it does not recognise, an audit of your site ends with “the site refused our crawler” and no report. Allowing the user agent is not always enough: a block by IP address or by country turns the crawler away whatever it calls itself. Allow both.
- User agent
- ChkWebAuditBot/1.0 (+https://chkweb.com/bot)
- Match on ChkWebAuditBot; the version number may change.
- IP address
- 193.32.177.88
- IPv4 only: the crawler does not connect over IPv6. This is the address of the server CheckWeb runs on, so it can change if we move to another one; this page always lists the current address. CheckWeb's one-page scanner connects from the same address.
If robots.txt keeps crawlers out, a group naming ours takes precedence over the wildcard group:
User-agent: ChkWebAuditBot
Allow: /It only runs when somebody with a CheckWeb account asks for an audit of your site. It does not crawl on a schedule or of its own accord, so letting it in lets those audits through and invites no others.
What it does, and what it does not
- One request every 250 milliseconds, at most, however many pages are queued. That is roughly four a second and it does not go up.
- Up to 100 pages per run today, and a run never goes past that.
- Only pages your site publishes: the home page, your sitemap, and links found on pages it already fetched. It does not guess at paths.
- GET and HEAD only. It never submits a form, never sends a cookie or an authorization header, and never signs in.
- It stays on your hostname. Links to other sites get a single HEAD request to check whether they are still alive.
- It obeys robots.txt, including a rule addressed to it by name.
Why it visited
CheckWeb is a website scanner. Somebody with an account asked for a full SEO audit of your site, and this crawler is what collects the pages that audit is built from. Every run is tied to that account, which is how we throttle and, where necessary, ban.
We limit how often any one site is crawled regardless of who asks, so a site cannot be audited repeatedly by different people to add up to real traffic.
Something looks wrong
If the crawler behaved differently from what is described here, we want to know. Write to hello@chkweb.com with the date and a few lines of your log, and include the requesting IP if you have it.