Indexability and crawling
How to fix pages that return errors
Some of your URLs answer with an error instead of a page. A 404 wastes the crawl budget Google gives your site; a 5xx makes it slow down or stop crawling for a while.
What it is, and what it costs
These pages return 4xx (not found, forbidden) or 5xx (server error) status codes. Readers hit a dead end, and search engines spend their limited crawl budget on URLs that lead nowhere. A run of 5xx errors during a crawl can make Google back off the whole site.
How to fix it
Restore or remove the pages that should exist. For anything that no longer exists, return a clean 404 or 410 rather than a server error, so search engines understand it is a removed page and not a broken server.
How to check your site
Crawl the site and list every URL that does not return 200. Google Search Console's Pages report groups them by error type.
Questions people ask
- Is a 404 always bad?
- No. A 404 for a page that no longer exists is correct. The problems are internal links pointing at it, and 5xx errors, which signal a broken server.
Does your site have this?
Check your site to find pages answering with an error.
See also
- How to fix canonical tag problemsA canonical tag names the real URL for a page when the same content is reachable more than one way. Getting it wrong is quieter than most SEO faults: nothing on the page looks broken, and its value quietly goes to the wrong address — or to one that cannot rank at all.
- How to fix XML sitemap problemsA sitemap is how search engines discover pages your own links do not lead to well, and how they learn something changed. Two things go wrong with it: there is not one, or the one you have lists URLs that do not work.
- How to fix a page accidentally set to “noindex”If a page you care about is nowhere in Google, not even when you search its exact title, a noindex tag is the first thing to rule out. It is one line, it overrides everything else you do for that page, and it usually got there by accident.
- How to fix a robots.txt that blocks your whole siteDisallow: / in your robots.txt is the fastest way to disappear from search. It asks every crawler to skip the whole site at once, and it is a common leftover from launch day.