Deduplication Strategies for Scraped Data #

Crawlers are duplicate machines. Pagination overlaps, canonical and tracking-parameter URLs point at the same page, re-crawls revisit unchanged content, and syndicated articles reappear verbatim across dozens of hosts. Without deliberate deduplication, a dataset inflates, aggregates skew, and — worst for compliance — you accumulate redundant copies of the same personal data. This technique, part of the Pipeline Storage, Deduplication & Monitoring section, lays out the full ladder of dedup methods: exact keys, cryptographic content hashing, near-duplicate detection with SimHash and MinHash, and probabilistic seen-sets backed by Redis or Bloom filters. It is written for anyone whose crawl touches the same content twice, which is every crawl that runs more than once. The goal is not merely a smaller table; it is a dataset where each stored record earns its place, which is exactly what the GDPR principle of data minimisation demands.

Core Principles & Duplication Taxonomy #

Deduplication only makes sense once you know which kind of duplicate you are fighting. There are three, and they call for different tools.

The cheapest duplicate to eliminate is the one you never fetch. URL-level dedup happens before a request is made, using a normalised URL and a shared seen-set — the pattern is covered end-to-end in distributed crawl scheduling and queues, where a Redis set keeps a fleet of workers from re-requesting the same page. Content-level dedup happens after extraction: exact hashing catches records that are byte-for-byte identical, while near-duplicate detection catches the messier and more common case of records that a human would call “the same” despite differing in a rotating ad slot, a view counter, or a session token. Each rung up this ladder costs more compute, so apply them in order and stop when the cheaper rung has done enough.

Implementation Steps #

1. Canonicalise before you compare anything #

Every dedup method downstream is only as good as the normalisation feeding it. Strip tracking parameters, lowercase the host, sort query keys, and remove fragments so trivially different URLs and records collapse to one identity.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

STRIP = {"utm_source", "utm_medium", "utm_campaign", "gclid", "fbclid", "ref"}

def canonical_url(url: str) -> str:
    parts = urlsplit(url.strip().lower())
    query = urlencode(sorted(
        (k, v) for k, v in parse_qsl(parts.query) if k not in STRIP
    ))
    # No fragment, sorted query, tracking params gone.
    return urlunsplit((parts.scheme, parts.netloc, parts.path.rstrip("/"), query, ""))

2. Compute a stable content fingerprint #

For exact-duplicate detection, hash the fields that define record identity with SHA-256. A stable canonical serialisation is essential — reorder a dict and a naive hash changes. The full production treatment, including field selection and collision handling, is in deduplicating records with content hashing.

import hashlib

def content_hash(record: dict, identity_fields: tuple[str, ...]) -> str:
    # Join with a control char that cannot appear in the values.
    canonical = "\x1f".join(str(record.get(f, "")) for f in identity_fields)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

Store this hash as a unique column. On write, an ON CONFLICT (content_hash) DO NOTHING upsert turns exact-duplicate suppression into a single idempotent statement — the same mechanism the structured data sinks guidance uses for replay safety.

3. Maintain a seen-set for O(1) membership checks #

To decide “have I processed this already?” without querying the main table, keep a seen-set of hashes. Redis is the pragmatic default for a distributed crawl; a Bloom filter trades a tiny false-positive rate for a fraction of the memory when the set runs into the hundreds of millions.

import redis

r = redis.Redis()

def first_time_seen(fingerprint: str, ttl_days: int = 30) -> bool:
    """Atomic check-and-set; True only on the first sighting."""
    key = f"seen:{fingerprint}"
    # SET NX returns None if the key already existed.
    added = r.set(key, 1, nx=True, ex=ttl_days * 86400)
    return bool(added)

The TTL matters for compliance: an unbounded seen-set is itself a growing store of identifiers, so expire entries once a re-crawl of that content is expected.

4. Detect near-duplicates with SimHash #

Exact hashing misses records that differ by one rotating byte. SimHash produces a fingerprint whose Hamming distance is small for similar content, letting you cluster “the same article, different wrapper” without an O(n²) comparison.

from simhash import Simhash

def near_dup(text_a: str, text_b: str, threshold: int = 3) -> bool:
    # Hamming distance <= threshold => near-duplicate. 3 is a common cut for prose.
    distance = Simhash(text_a).distance(Simhash(text_b))
    return distance <= threshold

For deduplicating a large corpus rather than comparing pairs, MinHash with an LSH index scales to millions of documents by bucketing candidates before exact Jaccard comparison — reach for it when SimHash’s pairwise model no longer fits the volume.

5. Decide upsert-vs-skip per duplicate class #

Not every duplicate should be discarded. An exact re-crawl of unchanged content should skip. A record whose key matches but whose content changed should upsert, updating the stored row and its crawled_at. A near-duplicate from a different source may warrant keeping both with a link, if provenance matters. Encode this policy explicitly rather than letting the database default decide.

Error Handling & Observability #

Dedup failures are silent by nature — the symptom is a dataset that is subtly too large or too small. Instrument it so the failure mode becomes visible.

Export metrics that make the dedup ratio observable: scraper_records_seen_total, scraper_duplicates_skipped_total{type}, and scraper_near_duplicates_flagged_total. The single most useful alert fires when the skip ratio collapses toward zero — that almost always means an unstable field crept into the identity set and every record now hashes unique.

{
  "event": "dedup_decision",
  "ts": "2026-07-05T09:31:07Z",
  "content_hash": "9f2c…a41",
  "decision": "skip_exact_duplicate",
  "source_url": "https://example.com/a",
  "seen_set": "redis",
  "correlation_id": "d19a-4c2b"
}

There is a subtle ordering dependency worth calling out: run the cheap rungs first and let them shrink the input to the expensive ones. Canonicalisation and the URL seen-set eliminate the bulk of duplicates before a single byte is fetched; exact content hashing then removes byte-identical records at write time for near-zero cost; only the survivors — records that are genuinely distinct byte-for-byte yet possibly redundant in meaning — reach the SimHash or MinHash stage, which is the most compute-hungry rung. Inverting this order, running near-duplicate detection over the raw firehose, wastes CPU comparing pages that a normalised URL would have collapsed for free. Measure the survivor count at each rung; if the expensive stage is still seeing most of your traffic, a cheaper rung upstream is misconfigured.

Compliance Boundaries #

Deduplication is a data-minimisation control, and GDPR’s minimisation principle expects you to hold no more personal data than necessary. Storing five copies of the same person’s record because five URLs surfaced it is a minimisation failure, not just wasted disk.

• Records containing personal data are deduplicated before persistence so no redundant copies accumulate.
• The seen-set has a TTL; it is not an unbounded permanent store of identifiers.
• Content hashes over personal data are treated as identifiers themselves and access-controlled accordingly.
• The dedup decision is logged with the record key The dedup decision is logged with the record *key*, never the sensitive fields.
• Deduplication is idempotent, so re-processing a batch cannot resurrect a record already erased under a deletion request.

That last point ties dedup to erasure: if a subject’s record was deleted but its fingerprint lingers in the seen-set, a re-crawl correctly skips re-adding it — but you must ensure erasure workflows also clear the corresponding curated rows. The handling of erasure and its interaction with scraped identifiers is covered in GDPR compliance for scraped personal data.

Common Mistakes #

1. Hashing an unstable field. Wrong: including a scraped_at timestamp or a view counter in the identity set, so every crawl produces a “new” record. Right: hash only fields that define the content’s identity.
2. Deduplicating after the write. Wrong: inserting everything and running a nightly de-dupe job over an already-bloated table. Right: check the seen-set or content hash before persisting.
3. Treating near-duplicates as exact. Wrong: expecting SHA-256 to catch articles that differ by one rotating ad token — it never will. Right: layer SimHash or MinHash on top for the fuzzy case.
4. An unbounded seen-set. Wrong: a Redis set that grows forever and eventually holds hundreds of millions of identifiers with no expiry. Right: TTLs sized to your re-crawl cadence, or a Bloom filter when memory is the constraint.
5. Failing closed on a dead seen-set. Wrong: skipping every record when Redis is down, silently dropping data. Right: fail open to a database unique constraint so nothing is lost.

Frequently Asked Questions #

Should I deduplicate URLs or content? #

Both, at different stages. URL-level dedup prevents wasted fetches and runs at the queue with a normalised URL and a seen-set, as described in distributed crawl scheduling and queues. Content-level dedup catches the cases URL dedup cannot — the same content served under different URLs or across different hosts — and runs after extraction. Neither replaces the other.

When do I need SimHash or MinHash instead of a plain hash? #

When “duplicate” means “substantially the same” rather than “byte-identical.” SHA-256 flips completely if a single character changes, so a page with a rotating timestamp, ad slot, or session token defeats it. SimHash gives you a similarity fingerprint for pairwise checks; MinHash with LSH scales that to deduplicating a large corpus. Use exact hashing first and only reach for these when residual near-duplicates are a real problem.

Is a Bloom filter safe for deduplication given its false positives? #

Yes, if you understand the trade. A Bloom filter can report “seen” for an item it has never seen (false positive) but never the reverse, so its only failure mode is occasionally skipping a genuinely new record. Sized for, say, a 0.1% false-positive rate, that is acceptable for most crawls and buys an enormous memory saving over storing every hash. When you cannot tolerate any missed record, back it with an exact store or a database unique constraint.

Related guides #

• Pipeline Storage, Deduplication & Monitoring — the parent section on what happens to scraped data after extraction.
• Deduplicating Records with Content Hashing — the SHA-256 fingerprinting pattern in production detail.
• Distributed Crawl Scheduling and Queues — URL-level dedup with a shared seen-set before fetch.
• Structured Data Sinks and Warehousing — turning a content hash into an idempotent upsert.
• GDPR Compliance for Scraped Personal Data — why minimisation makes deduplication a compliance control.