HTTP 403 Forbidden

`401 Unauthorized` means the request lacked valid credentials and the server will accept some. It must be accompanied by a `WWW-Authenticate` header naming the scheme, which is the client's instruction on how to retry.

`403 Forbidden` means the server is refusing regardless. It may know exactly who is asking and still say no, and there is no header telling the client how to succeed.

The practical difference: a 401 is a door with a lock, a 403 is a wall. A client that receives 401 has something to do next; a client that receives 403 does not.

Servers frequently return 403 where 404 would be more discreet, deliberately: answering 404 for a resource that exists but is not yours avoids confirming that it exists at all.

Bot management. A CDN or a web application firewall classifies the request as automated and refuses it, usually on the basis of the user agent string, the TLS fingerprint, or the absence of browser-like headers.

Rate limiting that returns 403 instead of 429. The refusal is temporary, but the code does not say so, and a second attempt a minute later frequently succeeds.

Geographic and network restrictions: content licensed by region, or an origin that only accepts traffic from its own CDN.

Hotlink protection on images, which refuses requests whose `Referer` does not belong to the site.

Directory rules: many servers answer 403 rather than 404 for a directory with no index file and listing disabled, which is a configuration default rather than a decision.

A page linked from your own navigation that returns 403 to ordinary visitors is broken, and it is worth confirming in a browser before treating it as a crawler artefact.

A CSS or JavaScript file returning 403 breaks the page for everyone. Subresources are requested with the same headers as the document, so a bot filter rarely explains it.

A 403 on a URL you expect search engines to index is a real indexing failure: a crawler that cannot fetch a page cannot index it, and repeated 403s lead to the URL being dropped.

A 403 on `/robots.txt` is worse than a missing file. Crawlers treat a persistently unreachable robots.txt as a reason to slow or stop crawling the whole host.

Open the URL in a normal browser session. If it loads, the refusal is aimed at automated clients and the page itself is fine.

Compare `curl -I https://example.com/path` against the same request with a browser user agent string. A different answer identifies the filter directly.

Check whether the pattern is per-URL or per-host. Scattered 403s across unrelated destinations point at your own outbound traffic being filtered; 403s concentrated on one host point at that host.

In Search Console, look at whether Googlebot is also refused. A 403 that affects only third-party checkers and not search engines is a reporting artefact; one that affects Googlebot is an indexing problem.