ChkWebAuditBot
ChkWebAuditBot/1.0 (+https://chkweb.com/bot)
If you found this page in your access log, that is exactly why it exists. ChkWebAuditBot crawls public pages to produce an SEO audit, the way Googlebot, Screaming Frog and Ahrefs do. Somebody asked for an audit of your site — often you, sometimes an agency or a developer working on it.
How to stop it
Add this to the robots.txt at the root of your site. We read robots.txt before anything else on every run, and a rule that names us is obeyed even when a wildcard group would allow the crawl.
User-agent: ChkWebAuditBot
Disallow: /It takes effect on the next run — there is nothing to wait for and nobody to email. To keep us out of one section instead, disallow that path rather than the whole site.
What it does, and what it does not
- One request every 250 milliseconds, at most, however many pages are queued. That is roughly four a second and it does not go up.
- Between 100 and 1000 pages per run, depending on the plan, and never more.
- Only pages your site publishes: the home page, your sitemap, and links found on pages it already fetched. It does not guess at paths.
- GET and HEAD only. It never submits a form, never sends a cookie or an authorization header, and never signs in.
- It stays on your hostname. Links to other sites get a single HEAD request to check whether they are still alive.
- It obeys robots.txt, including a rule addressed to it by name.
Why it visited
CheckWeb is a website scanner. Somebody with an account asked for a full SEO audit of your site, and this crawler is what collects the pages that audit is built from. Every run is tied to that account, which is how we throttle and, where necessary, ban.
We limit how often any one site is crawled regardless of who asks, so a site cannot be audited repeatedly by different people to add up to real traffic.
Something looks wrong
If the crawler behaved differently from what is described here, we want to know. Write to hello@chkweb.com with the date and a few lines of your log, and include the requesting IP if you have it.