Indexability and crawling
Do you need a robots.txt file?
There is no robots.txt on your site. Nothing breaks without one, but it is where you point crawlers at your sitemap and keep them out of admin and internal-search pages.
What it is, and what it costs
robots.txt is a small file at your domain root that guides crawlers. Missing it is not an error, but you lose an easy place to declare your sitemap and to steer crawlers away from low-value URLs that would otherwise eat crawl budget.How to fix it
Add a robots.txt at the root that allows crawling and points at your sitemap: User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
- WordPress: Yoast or Rank Math can create and edit it for you.
- Shopify: a robots.txt is generated automatically; you rarely need to add one.
How to check your site
Visit yoursite.com/robots.txt. A 404 means there is none.
Questions people ask
- Is a missing robots.txt urgent?
- No. It is a small improvement, not an emergency; add one when convenient.
Does your site have this?
Check your site to see whether a robots.txt is in place.
See also
- How to fix canonical tag problemsA canonical tag names the real URL for a page when the same content is reachable more than one way. Getting it wrong is quieter than most SEO faults: nothing on the page looks broken, and its value quietly goes to the wrong address — or to one that cannot rank at all.
- How to fix XML sitemap problemsA sitemap is how search engines discover pages your own links do not lead to well, and how they learn something changed. Two things go wrong with it: there is not one, or the one you have lists URLs that do not work.
- How to fix a page accidentally set to “noindex”If a page you care about is nowhere in Google, not even when you search its exact title, a noindex tag is the first thing to rule out. It is one line, it overrides everything else you do for that page, and it usually got there by accident.
- How to fix a robots.txt that blocks your whole siteDisallow: / in your robots.txt is the fastest way to disappear from search. It asks every crawler to skip the whole site at once, and it is a common leftover from launch day.