Broken link checker

Discovery: starting from the home page, it follows internal links to build the list of pages, bounded by the crawl budget and by robots.txt.

Extraction: from each page it collects the link targets — anchors, and the resources the page references.

Testing: it requests each target and records the status, following redirects and noting each hop.

Attribution: it records which page each broken link was found on, which is what turns a list of dead URLs into a list of edits.

No internal link resolving to 404 or 410.

Redirect chains of at most one hop, with internal links pointing at final destinations rather than at redirects.

External links either resolving or explicitly accounted for — a 403 from a site known to block automated clients is not a defect on your side.

And, importantly, a coverage figure you believe: the count of pages examined, and the reasons any were not.

Internal 404s, especially in clusters — one navigation component or one template pointing at a path that no longer exists.

Links whose targets differ only in case, which after a migration between systems is a recurring cause.

Redirect chains of three or more hops, which are slower and hide breakage at the end.

A page count far below the site's real size, which means the result describes a fraction of it.

Links behind a login are not reached: the crawl is anonymous.

Links inserted by scripts after load may not appear in the HTML the crawler reads.

Pages disallowed by robots.txt are not fetched, and the report says so rather than treating them as clean.

A 200 means reachable, not correct. A link pointing at the wrong existing page is invisible to any status-based check.

`availability.broken_links` reports broken targets with the page each was found on. `availability.status_classification` groups the status codes encountered. `transport.redirect_chain` reports the chains separately, since a link reached through four hops is a different defect from a dead one.

The crawl reports the pages it could not reach and the reason, so an empty finding reads as "nothing broken in what was covered".

A first run on a site of any age returns more rows than anyone wants to work through, and most of the rows are not worth the same attention. Triage before editing.

Internal 404s first, grouped by target. Twenty rows pointing at the same dead URL are one edit in a navigation component or one redirect, not twenty edits. This group is almost always where the real defects are.

Then internal links that resolve through a chain. Not urgent, and cheap to fix while the file is already open.

Then external 404s and 410s, checked in a browser before removal. A page that genuinely disappeared wants the link removed or repointed; often a replacement exists at a new address.

External 403s, 401s and 429s last, and expect most to be false alarms: these are servers declining an automated client, and a browser usually loads them without complaint.

The row that deserves suspicion rather than action is a whole external domain failing at once. That is far more likely to be the domain blocking the crawler than every one of its pages disappearing on the same day.