Compliant Web Scraping

Deduplicating records with content hashing #

The same entity reaches your pipeline through many doors: two URLs that render identical product data, a re-crawl that returns unchanged content, a paginated feed that overlaps the previous page. Storing all of them wastes space and skews analytics. Content hashing gives each record a compact fingerprint derived from its meaningful fields, so “have I already stored this exact content?” becomes a single index lookup. This page explains how to canonicalise a record, compute a stable SHA-256 over its normalised fields, store the digest, and skip on seen. It is a detail page under Deduplication Strategies for Scraped Data within the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

There are two distinct dedup questions and content hashing answers one of them precisely. Entity deduplication asks “are these two records the same thing?” — that is a natural-key or fuzzy-matching problem. Content deduplication asks “is this record byte-for-byte the same content I already have?” — that is a hashing problem. Content hashing is exact, cheap, and unambiguous: identical meaningful fields produce an identical digest, and any change produces a completely different one.

The trap is stability. A hash is only useful if the same logical record always produces the same digest. Serialise a Python dict without sorting keys and the hash flips between runs because dict ordering, whitespace, and float formatting all vary. Include a scraped_at timestamp in the hash and every re-crawl looks “new”. The entire technique lives or dies on a disciplined canonicalisation step that strips volatile fields and produces one deterministic byte string per logical record.

Step-by-Step Implementation #

  1. Select the semantic fields. Decide which fields define the content. Exclude volatile metadata — fetch timestamp, crawl id, proxy IP, response latency — that changes without the content changing.
  2. Normalise each field. Trim and collapse whitespace, lowercase where case is not significant, normalise Unicode to NFC, and format numbers and dates to a fixed representation. This is what makes " 9.90 " and "9.9" hash the same when they mean the same price.
  3. Serialise deterministically. Dump to JSON with sort_keys=True, no insignificant whitespace, and ensure_ascii=False, then encode as UTF-8. Sorted keys remove dict-ordering nondeterminism.
  4. Hash with SHA-256 over the canonical bytes to get a stable hex digest.
  5. Store and skip on seen. Keep the digest in a unique index (or a Redis set for hot lookups) and drop records whose hash already exists.
import hashlib
import json
import unicodedata

# Fields that must NOT enter the hash: they change without the content changing.
VOLATILE = {"scraped_at", "crawl_id", "proxy_ip", "response_ms", "etag"}

def _normalise_scalar(value):
    if isinstance(value, str):
        # NFC-normalise, strip, and collapse internal whitespace runs.
        text = unicodedata.normalize("NFC", value).strip()
        return " ".join(text.split())
    if isinstance(value, float):
        # Fixed formatting so 9.9 and 9.90 do not diverge.
        return format(value, ".4f")
    return value

def canonicalise(record: dict) -> bytes:
    """Produce one deterministic byte string per logical record."""
    cleaned = {
        k: _normalise_scalar(v)
        for k, v in record.items()
        if k not in VOLATILE and v is not None
    }
    return json.dumps(
        cleaned, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode("utf-8")

def content_hash(record: dict) -> str:
    """Stable SHA-256 hex digest over the record's normalised fields."""
    return hashlib.sha256(canonicalise(record)).hexdigest()

Wire the skip-on-seen check into the write path. A database unique index on the digest is the durable source of truth; a Redis set in front of it keeps the common “already seen” case off the database:

def store_if_new(conn, redis, record: dict) -> bool:
    digest = content_hash(record)
    # Fast path: SADD returns 0 if the member already existed.
    if redis.sadd("seen:content", digest) == 0:
        return False  # duplicate — skip silently
    with conn.cursor() as cur:
        # Durable guard: the unique index is authoritative if Redis is cold.
        cur.execute(
            "INSERT INTO record (content_hash, payload) VALUES (%s, %s) "
            "ON CONFLICT (content_hash) DO NOTHING RETURNING id",
            (digest, json.dumps(record)),
        )
        inserted = cur.fetchone() is not None
    conn.commit()
    return inserted

The Redis SADD is an optimisation, not the guarantee — the ON CONFLICT (content_hash) DO NOTHING on a UNIQUE index is what keeps the store correct even if the cache is flushed or a second worker races. This same digest feeds the conditional updates described in upserting scraped records into Postgres, so compute it once and pass it downstream.

Verification & Testing #

Assert the two properties that make the hash trustworthy — determinism and volatility-insensitivity:

def test_hash_is_stable_and_ignores_volatile_fields():
    a = {"sku": "A1", "title": "Widget ", "price": 9.9,
         "scraped_at": "2026-07-05T10:00:00Z", "proxy_ip": "10.0.0.1"}
    b = {"price": 9.90, "sku": "A1", "title": "Widget",   # reordered + reformatted
         "scraped_at": "2026-07-05T11:30:00Z", "proxy_ip": "10.0.0.9"}
    # Same content, different volatile metadata and key order → same digest.
    assert content_hash(a) == content_hash(b)

    c = {**a, "price": 10.0}                                # real change
    assert content_hash(a) != content_hash(c)

Confirm the collision surface is negligible with a git-style sanity check: SHA-256 over normalised fields makes an accidental collision astronomically unlikely, so any two records sharing a digest can be treated as identical content. Measure the dedup rate in production by comparing SADD hits to total records — a sudden drop toward zero usually means a volatile field leaked into canonicalise.

Compliance & Operational Guardrails #

  • A hash of personal data is still personal data. A SHA-256 digest over a name and address is a pseudonym, not anonymisation — it is reversible by anyone who can re-hash a guess. Keep hashed personal fields inside the same retention and access controls as the raw record, per GDPR compliance for scraped personal data.
  • Version your canonicalisation logic. If you change which fields or normalisation rules feed the hash, store a hash_version alongside the digest so old and new digests are never compared as equals.
  • Make dedup auditable. Log the digest and the skip decision so an audit can show why a record was or was not stored, without retaining the duplicate payload itself.

Common Mistakes #

  1. Hashing the raw serialised dict. Unsorted keys, stray whitespace, and float formatting make the digest unstable across runs, so nothing ever matches. Canonicalise first.
  2. Including volatile fields. A scraped_at or proxy_ip inside the hash makes every re-crawl look new, defeating the entire purpose.
  3. Trusting the cache as the source of truth. A Redis set alone loses all history on a flush; back it with a UNIQUE index and ON CONFLICT DO NOTHING.

Frequently Asked Questions #

Why SHA-256 rather than a faster hash like MD5 or a CRC? #

CRCs and short hashes have high collision rates at scale — with millions of records you will get accidental collisions that silently drop distinct content. SHA-256’s 256-bit space makes accidental collisions effectively impossible, and the CPU cost is trivial next to network and parsing time. Non-cryptographic hashes such as xxHash are acceptable only for a first-pass filter that is confirmed by an exact comparison.

How do I dedup near-duplicates that differ by a word or a timestamp in the body? #

Content hashing is exact by design and will not catch near-duplicates — that is a feature, because it never merges records that genuinely differ. For fuzzy matching (say, the same article republished with minor edits), use a similarity technique such as MinHash or SimHash as a separate stage; keep exact content hashing as the fast, unambiguous first line of defence.