robots.txt blocking pages that should be crawled

Rules are grouped by `User-agent`. A crawler obeys the most specific group that names it, and ignores the others entirely — so a `Disallow` under `User-agent: *` does not apply to Googlebot if a `User-agent: Googlebot` group exists.

Paths are matched by prefix. `Disallow: /admin` blocks `/admin`, `/administration` and `/admin-tools`, which is usually more than intended. `Disallow: /admin/` blocks only the directory.

`*` matches any sequence of characters and `$` anchors the end of the path. `Disallow: /*.pdf$` blocks PDFs; `Disallow: /*?` blocks every URL with a query string, which is a frequent accident.

Where both an `Allow` and a `Disallow` match, the more specific rule wins — the one with the longer path. This is how a directory is blocked with one file inside it permitted.

An empty `Disallow:` means nothing is disallowed. A `Disallow: /` blocks the entire site, and that single character is the most consequential typo in the file.

A blocked URL can still appear in search results. Google documents that it may index a URL it has never fetched, based on links pointing at it, and show it without a description.

Worse, blocking prevents removal: a page carrying `noindex` that is also disallowed will never have that directive read. The two mechanisms work against each other.

So the rule is simple. To stop crawling, use `robots.txt`. To stop indexing, allow crawling and serve `noindex`. To protect something, use authentication — `robots.txt` is a public file and listing a path in it advertises the path.

Blocking `/assets/`, `/static/` or `/wp-includes/` stops the crawler fetching CSS and JavaScript. Google renders pages to understand them, and a page whose stylesheet it cannot load is assessed on a broken rendering — including for mobile-friendliness.

Blocking `/*?` removes every URL with a query string, which on many sites is the entire product catalogue.

Blocking an API path that the page calls to render its content leaves the rendered page empty.

Google's guidance is explicit that CSS and JavaScript needed to render a page should not be blocked. If crawl volume on assets is the concern, caching headers are the tool, not exclusion.

Fetch the file directly: `curl https://example.com/robots.txt`. It must be at the root of the host, and each subdomain has its own — `robots.txt` on the apex does not govern `shop.example.com`.

Read the groups before the rules. Find which `User-agent` group applies to the crawler you care about, and read only that group.

In Search Console, the robots.txt report shows the file Google last fetched and whether it parsed, and the URL inspection tool reports when a specific URL is blocked.

Test the paths that matter rather than the file as a whole: a deep product URL, an asset URL, a paginated URL, a URL with parameters.

Make every `Disallow` a directory or an anchored pattern rather than a bare prefix, so it blocks what it names and nothing adjacent.

Remove rules covering CSS, JavaScript, fonts and images the pages need to render.

Replace `robots.txt` entries meant to hide pages with `noindex` on those pages, and with authentication where the content is genuinely private.

Declare the sitemap in the file — `Sitemap: https://example.com/sitemap.xml` — which is the one positive instruction it carries.

Keep the file small and readable. Most sites need a handful of lines, and a long file is usually accumulated rules nobody can account for.

`seo.robots_txt` fetches the file, reports whether it exists and parses, and reports the rules it contains together with the crawled URLs those rules affect.

The crawl itself reports the pages it could not fetch and the reason, so a page excluded by `robots.txt` appears as an explicit exclusion rather than silently missing from the results.

`seo.sitemap` covers the other half: whether the sitemap the file advertises exists and lists URLs that are actually crawlable.