Indexability and crawling
How to fix a robots.txt that blocks your whole site
Disallow: / in your robots.txt is the fastest way to disappear from search. It asks every crawler to skip the whole site at once, and it is a common leftover from launch day.
What it is
robots.txt sits at the root of your domain and tells crawlers where they may go. A single line, Disallow: / under User-agent: *, closes the entire site to every search engine.Why it costs you
Blocked pages cannot be crawled, so over time they drop out of results entirely. Sites often ship this way on purpose while they are being built, then go live without anyone remembering to remove the block. The homepage source looks fine; the damage is one file away, at
yoursite.com/robots.txt.How to fix it
User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
- WordPress: the usual cause is Settings → Reading → “Discourage search engines,” which makes WordPress serve a blocking robots.txt. Untick it. If you edit robots.txt through a plugin (Yoast, Rank Math), remove the
Disallow: /there. - Shopify: Shopify generates robots.txt from
robots.txt.liquid. If someone added a site-wide disallow, edit that template to remove it. - Any platform / hand-coded: open
yoursite.com/robots.txt. Delete theDisallow: /line. A safe default allows everything and points at your sitemap:User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
How to check your site
Visit yoursite.com/robots.txt in a browser. If you see Disallow: /, you are blocking everything. Google Search Console’s robots.txt report also flags it.
What this looks like in practice
An agency relaunched a client on a new host that carried over the staging robots.txt. Rankings fell off over three weeks before anyone thought to check the file. Removing one line, plus a re-crawl request, recovered them.
Questions people ask
- Is robots.txt the same as noindex?
- No. robots.txt stops crawling; noindex stops indexing. A page blocked in robots.txt can still show as a bare link in results, which is its own confusing problem, so never rely on robots.txt to hide something.
- How fast does it recover?
- Once the block is gone and Google re-crawls, pages return over days to weeks.
Does your site have this?
Run a free check to confirm your robots.txt is not shutting crawlers out.
See also
- How to fix canonical tag problemsA canonical tag names the real URL for a page when the same content is reachable more than one way. Getting it wrong is quieter than most SEO faults: nothing on the page looks broken, and its value quietly goes to the wrong address — or to one that cannot rank at all.
- How to fix XML sitemap problemsA sitemap is how search engines discover pages your own links do not lead to well, and how they learn something changed. Two things go wrong with it: there is not one, or the one you have lists URLs that do not work.
- How to fix a page accidentally set to “noindex”If a page you care about is nowhere in Google, not even when you search its exact title, a noindex tag is the first thing to rule out. It is one line, it overrides everything else you do for that page, and it usually got there by accident.
- How to fix redirect chains on your siteA redirect that points at another redirect makes the reader wait through every hop, and a bit of the value from the original link leaks away at each one. Chains build up quietly over years of moving pages around.