HTTP 4xx client error status codes

404 says there is nothing at this address now, without saying whether there ever will be. 410 says the resource is gone and that the condition is likely permanent — the honest answer for a page removed on purpose.

401 and 403 are refusals, not absences. 401 asks for credentials and names the authentication scheme; 403 refuses even if credentials would be supplied. A page behind a login answers one of them, and so does a firewall that does not like the client.

429 is a rate limit. It is the only code of the class that is temporary by design, and the Retry-After header tells a well-behaved client how long to wait.

A link check sends HEAD first, because it needs the status and not the page. When the answer is 405 or 501 — servers that do not implement HEAD — it repeats the request with GET and judges that answer instead. Any final status of 400 or above marks the link as broken, and a 404 is reported with a higher severity than the other codes.

A crawled page is handled more finely. 401 and 403 are not recorded as defects of the page: the page exists and refused an automated visitor, so it is listed as restricted, with the kind of protection met, and not analysed. A login or a challenge is never bypassed.

A 401, 403, 406 or 429 is first retried once with a standard browser identity, because some servers refuse bot user agents rather than the request. That retry is one ordinary, unauthenticated request with standard headers.

408, 425 and 429 are treated as temporary. By default a URL gets up to three attempts in total, with a backoff that honours Retry-After but never waits more than four seconds, so a site asking for a long pause gets a fast, reported fallback rather than a stalled scan.

The entry page is classified separately: a 4xx on the address the scan starts from is reported by `availability.status_classification` as a client error, a different state from a server error or a DNS failure.

RFC 9110 does not hold the whole class. 428, 429 and 431 were added by RFC 6585, 451 by RFC 7725, 425 by RFC 8470, and 423 and 424 by the WebDAV specification. All of them are in the IANA registry, which is the authoritative list of what a status code may mean.

RFC 9110 also renamed two codes: 413 is now Content Too Large and 422 is Unprocessable Content. Older software and documentation still use the earlier reason phrases; the number is what matters on the wire.