Website problems

Near-duplicate pages

Near-duplicate pages are two or more URLs whose main content is substantially the same, differing only in a value, a filter or a phrase. Search engines group such pages and index one of them, which may not be the one you would have chosen. There is no penalty for this — Google has said repeatedly that duplicate content does not incur a manual action — but the pages dilute each other's signals and the choice of which survives stops being yours.

The four patterns

Parameters and sort orders. `?sort=price`, `?utm_source=…`, `?sessionid=…` each produce a distinct URL serving the same content. This is the most common and the easiest to fix.

Templated variants. A page per city or per colour, generated from one body of text with a value substituted. These are near-duplicates by construction, and they are also usually thin.

Structural duplicates. The same item reachable under two categories, an article at both a dated path and a slug path, or a page served with and without a trailing slash.

Syndication. The same article published on several domains, including your own partners', where the question is which copy search engines should credit.

What search engines actually do

They cluster the URLs they judge equivalent and pick one to index — Google calls it the canonical of the group. The others remain crawlable and are generally not shown.

The choice uses several signals: the `rel=canonical` you declare, redirects, internal links, sitemap entries and which URL looks cleaner. A canonical tag contradicted by every internal link is a hint the engine may decline.

Search Console reports the outcome directly, as 'Duplicate, Google chose different canonical', naming the URL it picked. That report is the only place you learn what actually happened.

There is no penalty attached to this. The cost is the crawling spent on copies and the loss of control over which address represents the content.

What is not a near-duplicate

Pagination. Page two of a listing contains different items from page one, so the two are not duplicates and should each be canonical for themselves. Pointing page two at page one is a common misreading of this problem and it asks search engines to discard everything page two contains.

A page and its printable or AMP variant, where one is a rendering of the other. Those want a canonical, which is exactly the case the tag was designed for.

Pages sharing a template but not a body. Two articles using the same layout are not duplicates; the boilerplate is not the content.

Translations. A page and its French version cover the same subject in different languages, and `hreflang` — not `rel=canonical` — is what relates them. Canonicalising a translation to the original removes it from results in its own language.

How to check it yourself

Take one page and try the variants: append a meaningless parameter, toggle the trailing slash, switch the host, switch the protocol. Each should either redirect or declare the same canonical URL.

Compare titles and meta descriptions across the site. Groups sharing both are the fastest signal that the bodies are also close.

For templated sets, read two pages side by side and count the sentences that differ. If a handful of substituted values is the whole difference, they are near-duplicates whatever their word count.

In Search Console, the Page indexing report's duplicate categories name both the excluded URL and the one Google chose.

How to fix each pattern

Parameters: give every page a self-referencing absolute canonical. The tracking-parameter case disappears entirely, at no cost.

Structural duplicates: pick one URL and redirect the others with 301. A redirect is a stronger and clearer statement than a canonical, and it is the right tool when the duplicate address should not be reachable.

Templated sets: this is an editorial decision, not a markup one. Either give each page substance that is genuinely specific to it, or collapse the set into one page that covers the subject and redirect the rest.

Syndication: cross-domain canonical tags are supported and are the standard mechanism for telling search engines which copy is the original.

What does not work is canonicalising a set of thin pages to each other and considering it handled. The markup describes the duplication; it does not remove the reason the pages were thin.

How VeriFixScan detects it

`structure.duplicate_content` compares the main content of the pages the crawler reached and reports groups whose bodies are substantially the same, with the URLs in each group.

`seo.duplicate_titles` and `seo.canonical` answer the adjacent questions: whether the duplicated pages also share a title, and whether they already declare a canonical URL.

`domain.url_variants` covers the protocol, host and trailing-slash forms, which is where structural duplication usually starts.

The comparison runs over the pages actually crawled, so an empty finding means no duplicates were found in what was covered — which the report states rather than implying a clean site.

Frequently asked questions

Is duplicate content penalised?
No. Google has stated there is no duplicate-content penalty of that kind. What happens is clustering: the engine indexes one URL of a group and generally does not show the others.
Canonical tag or redirect?
Redirect when the duplicate address should not be reachable at all. Canonical when it must keep working — a filtered view, a tracking parameter — but should not compete in the index.
Does changing a few words make pages distinct?
Not usefully. If a substituted value is the only difference, the pages remain equivalent to a reader, and a reader is what the clustering is approximating.

Sources

Related

VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.

Scan a website