Indexability and crawling
Links pointing at pages blocked by robots.txt
Your robots.txt tells crawlers to stay out of some pages, and your own site links to them. For a login, a cart, or an admin area that is exactly right and there is nothing to do. It is worth a glance because the same rule occasionally catches a section nobody meant to hide.
What it is, and what it costs
Disallow that is broader than intended and quietly keeps a real content section out of search.How to fix it
Read the list. If everything on it is meant to be private, this is working as designed. If something should be found in search, remove the Disallow rule that covers it in robots.txt.
How to check your site
Compare your internal links against your robots.txt rules; anything linked but disallowed shows up here. Google Search Console reports pages "Indexed, though blocked by robots.txt," which is the confusing middle state to avoid.
Questions people ask
- Is linking to a blocked page bad?
- Not in itself. It is only worth acting on if the blocked page is one you actually want ranking.
Does your site have this?
Check your site to see which linked pages your robots.txt is blocking.
See also
- How to fix canonical tag problemsA canonical tag names the real URL for a page when the same content is reachable more than one way. Getting it wrong is quieter than most SEO faults: nothing on the page looks broken, and its value quietly goes to the wrong address — or to one that cannot rank at all.
- How to fix XML sitemap problemsA sitemap is how search engines discover pages your own links do not lead to well, and how they learn something changed. Two things go wrong with it: there is not one, or the one you have lists URLs that do not work.
- How to fix a page accidentally set to “noindex”If a page you care about is nowhere in Google, not even when you search its exact title, a noindex tag is the first thing to rule out. It is one line, it overrides everything else you do for that page, and it usually got there by accident.
- How to fix a robots.txt that blocks your whole siteDisallow: / in your robots.txt is the fastest way to disappear from search. It asks every crawler to skip the whole site at once, and it is a common leftover from launch day.