What happens if robots.txt is unreachable?

200 with content: the rules are parsed and applied. This is the ordinary case.

404 or another 4xx: the crawler concludes there are no rules and crawls freely. Nothing is lost, and a site with no robots.txt is a correctly configured site.

5xx, a connection failure or a timeout: the crawler cannot know what the rules are. Google's documentation states it will stop crawling the site and retry the file, on the principle that acting without knowing the rules risks fetching something forbidden.

A redirect is followed, within limits, and a redirect chain that never resolves behaves like the third case.

The gap between the second and third outcome is the whole reason this matters: one is harmless and the other is a crawl outage.

A `Disallow: /` blocks crawling of the paths you named and leaves the rest of the host reachable.

A failing robots.txt suspends crawling of everything, because the suspension is about the unknown rather than about a rule.

The effect is temporary and recovers when the file answers again, but recovery is on the crawler's retry schedule, not yours.

Sustained failure over days is the case that reaches the index: pages stop being refreshed, and eventually some are dropped.

None of this is visible on the site itself. Every page continues to serve perfectly to visitors while crawling quietly stops.

The application serves robots.txt rather than the web server, so an application outage takes the file with it — precisely when crawlers most need to know the rules.

A firewall or bot filter classifies the crawler as automated and answers 403 or 503 for the file specifically.

A rate limit triggered by a burst of crawling refuses the next request, including the robots.txt fetch.

A CDN configuration that caches an error response and then serves it for its full lifetime.

A maintenance mode rule applied to every path, which is correct for pages and wrong for this one.

`curl -sI https://example.com/robots.txt` gives the status you receive. Repeat it with a crawler-like user agent and with the default `curl/` string, since a bot filter can answer differently.

Search Console's robots.txt report shows the status Google last observed and when, which is the answer that counts for Google specifically.

Check it on every host that serves pages, including `www` and any subdomain: the file is per host and an application subdomain frequently has no rule for it at all.

Monitor it the way you monitor a page. A file that returns 200 today and 503 next Tuesday is exactly the failure this section describes.

If you have no rules, serve an empty file or a `User-agent: *` / `Allow: /` pair with 200. It is unambiguous and cannot be mistaken for an error.

Serve it from the web server or the CDN rather than the application, so it survives an application outage.

Exclude it from maintenance-mode rules and from bot filtering explicitly.

Keep it small. Google documents a 500 kibibyte limit and ignores content past it.

Include the `Sitemap:` line, which is read independently of the crawl rules and costs nothing.

VeriFixScan fetches robots.txt during a crawl and `seo.robots_txt` reports whether it was reachable and what it permits.

An unreachable file is reported as such rather than as an absent one, because the two mean opposite things to a crawler.

The scan observes the file from one client at one moment, so a filter that refuses only some clients may not be reproduced.

Where the site is behind a bot filter, comparing the scan result against Search Console's observation is what separates a real outage from a filter aimed at automated traffic.