robots.txt vs noindex

robots.txt is a request not to fetch. A compliant crawler reads the file before requesting a URL and skips the ones the rules forbid. Nothing about the page is read, including its content and its directives.

noindex is an instruction not to index, served either as a robots meta tag in the HTML head or as an `X-Robots-Tag` HTTP header. The crawler must fetch the page to see it.

So one operates before the request and the other inside the response. That ordering is the whole relationship between them.

A blocked URL can still appear in results, without a description, on the strength of links pointing at it — Google documents this explicitly.

Block a URL in robots.txt and also serve noindex, and the crawler never reaches the second instruction.

The page therefore remains eligible for indexing from external signals, which is the opposite of what both directives were meant to achieve.

Search Console reports this combination as an anomaly rather than leaving you to guess, and it is one of the more common findings on sites that have tried to clean up their index.

The sequence that works: allow crawling, serve noindex, wait for the page to be re-crawled and dropped, and only then block it in robots.txt if you also want to stop the requests.

Blocking first is the mistake, and it leaves the page indexed indefinitely with no description.

Infinite or near-infinite URL spaces: a calendar with a next-month link, faceted navigation where every filter combination is a URL, internal search results.

Endpoints with no value in search that still cost requests — export links, print views, sort parameters.

Anywhere the goal is to reduce fetching rather than to change what appears in results.

It is the right tool when you do not care whether the URL is indexed and do care that it stops being requested.

Pages that must not appear in results: internal search pages, thin filtered listings, staging environments, checkout confirmations, printer-friendly duplicates.

Anything already indexed that you want removed. This is the only one of the two that removes a page.

Non-HTML files, using the `X-Robots-Tag` header, since a PDF or an image has no head to put a meta tag in.

It is the right tool when the page must remain fetchable — because you want the directive read, or because links on it should still be followed.

Duplicates that should consolidate rather than disappear. A canonical tag transfers the page's signals to the preferred URL; noindex discards them.

Anything genuinely private. Both mechanisms are advisory and the page stays fully accessible to anyone with the URL. Authentication is the only answer.

Content you want gone immediately: removal tools in Search Console are faster than waiting for a re-crawl, and are the documented path for urgent cases.

Pages that are simply low quality. Neither directive improves a page, and a search engine declining to index a thin page is a judgement rather than a fault to suppress.

`curl -sI https://example.com/page | grep -i x-robots` finds the header form, and `curl -s https://example.com/page | grep -i 'name="robots"'` finds the meta tag. Check both; a clean head does not rule out a header.

Fetch robots.txt separately and test the URL against its rules rather than reading them.

VeriFixScan reports `seo.noindex` for every crawled page and `seo.robots_txt` for the file, which surfaces the contradiction where a blocked path also carries a directive nobody can read.

For pages rendered in the browser, check the rendered DOM as well: a directive injected by a script is honoured when the page is rendered and is absent from the served HTML.