Sitewide Duplicate Content Finder

How This Sitewide Duplicate Content Audit Works

1. Build a bounded inventory. Start a same-host link crawl or paste the important URLs from a sitemap, CMS, analytics, Search Console, or another crawler. Inventory mode is the safer choice for orphaned pages.
2. Extract search signals. For each server-rendered HTML response, the audit records the final URL, status, title, first H1, canonical, robots directives, indexability, and readable content blocks.
3. Measure template repetition. Navigation, headers, footers, scripts, forms, and similar chrome are removed first. Substantive blocks repeated across at least 40% of the successful crawl sample and at least three pages are measured as boilerplate and removed from a second, core-text representation.
4. Compare both content views. The complete extracted main text prevents genuinely copied sections from disappearing merely because they recur across several pages; the de-boilerplated core view reduces template noise. Exact fingerprints and three-word shingle Jaccard and containment scores are calculated on both, and a strong result from either view can surface a pair.
5. Cluster and prioritize. Qualifying pairs form transparent single-link content families. Each cluster reports direct-pair coverage and its weakest qualifying overlap, proposes a preferred-page candidate, and receives a context-aware consolidation or differentiation recommendation.
6. Verify before changing URLs. Filter the full inventory, inspect canonical and indexing signals, export CSV, then add backlinks, traffic, conversions, and business value before making a final decision.

Why Sitewide Context Is Better Than a Pairwise Score

Two pages can look similar for very different reasons. Product variants may deliberately share specifications. Location pages may repeat service information but still need meaningful local evidence. Tracking parameters may produce exact URL copies. A sitewide audit exposes the pattern, surrounding signals, and scale.

Google describes redirects and rel="canonical" as strong canonicalization signals, while sitemap inclusion is weaker. Signals can reinforce one another, but Google still chooses the representative URL it considers best. That is why this tool reports canonical alignment rather than pretending a similarity percentage determines indexation. See Google’s canonicalization guidance.

• Consolidate signals: retire obsolete copies and point links, sitemaps, canonicals, and redirects at one useful destination.
• Protect distinct intent: keep pages that answer meaningfully different needs, then make their headings, evidence, copy, and internal links genuinely distinct.
• Expose template risk: repeated legal text or product chrome may dominate thin pages even when the URLs are not literal copies.
• Catch conflicting directives: near-identical pages with self-canonicals, cross-canonicals, noindex, or non-200 responses deserve different fixes.

How to Choose: Redirect, Canonical, Noindex, or Rewrite

Limits You Should Know Before Acting

This is a focused free audit, not an unlimited rendering crawler. It reviews at most 50 public, same-host HTML pages and stops after roughly 42 seconds. It does not render JavaScript, log in, read private Search Console data, inspect backlinks, or know which page earns revenue. Similarity thresholds surface candidates; they do not diagnose a penalty.

For the best coverage, combine a link crawl with a separate URL inventory drawn from XML sitemaps, CMS exports, analytics landing pages, Search Console pages, server logs, and backlink tools. Run the same sample before and after consolidation so changes are comparable.