How do I find every page on my website?

A crawl follows internal links from the home page. It misses pages nothing links to, pages reachable only through forms or search, and anything excluded by the robots file.

The XML sitemap lists what a person or a plugin declared. It misses pages the generator does not know about and frequently still lists pages that were removed.

The content system's export lists what was published there. It misses anything served outside it — static files, landing pages built elsewhere, an old section on another stack.

Server or CDN logs record every URL that was requested and answered. They miss pages nobody visited in the period the logs cover, and they include plenty of URLs that never existed.

Analytics adds pages real visitors reached, minus everyone who blocked the script.

In the sitemap but not reached by the crawl: orphan pages, which nothing on the site links to.

Reached by the crawl but not in the sitemap: a sitemap generator that is incomplete or out of date.

Requested in the logs but in neither: old URLs still receiving traffic, often from external links, which need a redirect rather than a 404.

In the content system but answering with an error: pages published and then broken by a template or a routing change.

Each of these is a list of specific URLs to act on, which is the practical output of the exercise.

Normalise every URL the same way before comparing — scheme, host, trailing slash, letter case, and parameters you know to be irrelevant.

Otherwise the same page appears as four different entries and every comparison is noise.

Record the status code each URL returns now, since an inventory of addresses without their current state cannot tell a live page from a dead one.

Record the canonical each page declares, so duplicate addresses collapse into the page they belong to.

Keep the source of each URL alongside it, since which sources a URL appears in is the finding.

Before a migration, where every URL that exists today needs a destination tomorrow. A redirect map built from the crawl alone misses the pages the crawl could not reach.

Before a redesign, to know what content exists before deciding what to keep.

After an acquisition or a handover, when nobody remaining knows what the site contains.

To find pages that should not be public — old campaigns, test pages, forgotten drafts — which are frequently the ones only the logs know about.

For a content review, where pages that receive no traffic and no links are candidates for improvement or removal.

It is the one source that shows the site as a visitor and a search engine find it, by following the links actually present.

It reports the pages it reached, the status each returned and the links that lead nowhere.

It reads the sitemap and the robots file, so the declared set and the excluded set can be compared with the discovered one.

It is bounded by a budget, so on a large site it is a sample and says so in its coverage figure.

It cannot see logs or the content system, which is why it is one input to the inventory rather than the inventory.