Normalising Text Before Hashing for Dedup #

Two strings that a reader would call identical can hash differently for half a dozen reasons: a non-breaking space, a decomposed accent, a typographic apostrophe, a trailing newline. Every one of those turns a duplicate into a distinct record. Normalisation is the step that makes a hash mean “the same content” rather than “the same bytes”, and getting it right is a prerequisite for every other deduplication technique. This guide covers it, as part of deduplication strategies for scraped data in the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

Scraped text arrives carrying artefacts of how it was rendered rather than what it says. Six categories account for nearly all spurious hash differences.

Every one of these turns a duplicate into a distinct stored record.

Unicode composition. é can be one code point or two (e plus a combining acute). Both render identically and neither is wrong.

Invisible characters. Non-breaking spaces, zero-width joiners, soft hyphens and directional marks survive a naive strip() and are invisible in a terminal.

Typographic substitution. Sites replace ' with ’ and - with – at render time, so the same source text differs between templates.

Whitespace. Indentation and line wrapping in the markup become runs of spaces and newlines in the extracted text.

Case. Whether case matters is a domain decision — for a title it usually does not, for a product code it may.

Entity residue. & decoded once is &; decoded twice from double-encoded source it is something else entirely.

The consequence is not merely a bloated table. Deduplication that fails silently means the same personal data is stored several times, which is a data-minimisation problem as much as a storage one.

Step-by-Step Implementation #

1. A single, versioned normalisation function #

A fixed field tuple means adding a field does not change existing keys.

import re
import unicodedata

NORMALISATION_VERSION = "text-norm/3"

# Characters that look like a space and are not.
SPACE_LIKE = dict.fromkeys(map(ord, "        "
                                    "       　"), " ")
# Characters with no visual presence at all.
INVISIBLE = dict.fromkeys(map(ord, "‌‍⁠"), None)
# Typographic variants folded to their ASCII equivalents.
TYPOGRAPHIC = {
    ord("‘"): "'", ord("’"): "'", ord("‚"): "'", ord("‛"): "'",
    ord("“"): '"', ord("”"): '"', ord("„"): '"',
    ord("–"): "-", ord("—"): "-", ord("−"): "-",
    ord("…"): "...",
}

def normalise_text(value: str | None, *, casefold: bool = True) -> str:
    """Canonical form for hashing. Deterministic, versioned, and applied once."""
    if not value:
        return ""
    text = unicodedata.normalize("NFC", value)     # compose accents consistently
    text = text.translate(SPACE_LIKE)
    text = text.translate(INVISIBLE)
    text = text.translate(TYPOGRAPHIC)
    text = re.sub(r"\s+", " ", text).strip()
    return text.casefold() if casefold else text

NFC rather than NFD: composed form is what most sources emit and what most other systems expect, so choosing it minimises surprises at boundaries. casefold() rather than lower() handles cases lower() does not — German ß folds to ss, which is what a case-insensitive comparison should do.

Note what this function does not do. It does not use NFKC, which also rewrites ligatures, superscripts and full-width characters. That is sometimes desirable and sometimes destructive — x² becoming x2 changes meaning in a specification — so the aggressive form belongs behind an explicit flag rather than in the default path.

2. Per-field policies, because fields differ #

from dataclasses import dataclass

@dataclass(frozen=True)
class FieldPolicy:
    casefold: bool = True
    strip_punctuation: bool = False
    collapse_digits: bool = False        # for identifiers with variable formatting

POLICIES = {
    "title":       FieldPolicy(casefold=True),
    "description": FieldPolicy(casefold=True),
    "sku":         FieldPolicy(casefold=True, strip_punctuation=True),
    "isbn":        FieldPolicy(casefold=True, strip_punctuation=True, collapse_digits=True),
    "author_name": FieldPolicy(casefold=False),      # case can be meaningful in a name
}

def normalise_field(name: str, value: str | None) -> str:
    policy = POLICIES.get(name, FieldPolicy())
    text = normalise_text(value, casefold=policy.casefold)
    if policy.strip_punctuation:
        text = re.sub(r"[^\w\s]", "", text)
    if policy.collapse_digits:
        text = re.sub(r"\s+", "", text)
    return text

Per-field policies prevent the two failure directions. Too little normalisation and 978-0-13-235088-4 differs from 9780132350884; too much and two genuinely different product codes collapse into one. Making the policy explicit per field forces the decision to be made rather than inherited.

3. Build the hash from a canonical serialisation #

import hashlib
import json
from decimal import Decimal

IDENTITY_FIELDS = ("title", "sku", "brand", "price_minor", "currency")

def dedup_key(record: dict) -> tuple[str, str]:
    payload = {}
    for field in IDENTITY_FIELDS:
        value = record.get(field)
        if isinstance(value, str):
            payload[field] = normalise_field(field, value)
        elif isinstance(value, float):
            raise TypeError(f"{field}: floats do not hash reproducibly; use Decimal")
        elif isinstance(value, Decimal):
            payload[field] = str(value.normalize())
        else:
            payload[field] = value

    encoded = json.dumps(payload, sort_keys=True, ensure_ascii=False,
                         separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest(), NORMALISATION_VERSION

Four properties make the key reproducible: a fixed field tuple so adding a field does not change existing keys; sorted keys so dictionary ordering is irrelevant; ensure_ascii=False so the encoding is UTF-8 rather than escape sequences; and an outright refusal of floats, which serialise differently across platforms and versions.

4. Version the normalisation and migrate deliberately #

def needs_rehash(stored_version: str) -> bool:
    return stored_version != NORMALISATION_VERSION

def migrate_keys(store, batch: int = 10_000) -> dict:
    """Compute the new key alongside the old; switch only after comparing collapse rates."""
    stats = {"examined": 0, "changed": 0, "new_collisions": 0}
    for record in store.iter_where(norm_version_ne=NORMALISATION_VERSION, limit=batch):
        new_key, version = dedup_key(record)
        stats["examined"] += 1
        if new_key != record["dedup_key"]:
            stats["changed"] += 1
            if store.exists_key(new_key):
                stats["new_collisions"] += 1
        store.set_pending_key(record["id"], new_key, version)
    return stats

new_collisions is the number to look at before switching. A normalisation change that merges two thousand records which were previously distinct may be exactly right — or it may mean the new rule is too aggressive. Computing both keys and comparing before switching is what turns that into a decision rather than a discovery.

Verification & Testing #

Aggressive normalisation on prose destroys meaning for no dedup benefit.

NORMALISATION_CASES = [
    ("café",              "café",            True),   # NFC vs NFD
    ("hello world",       "hello world",      True),   # non-breaking space
    ("it's fine",         "it’s fine",        True),   # typographic apostrophe
    ("  spaced  out  ",   "spaced out",            True),
    ("softhyphen",  "softhyphen",            True),
    ("Title Case",        "title case",            True),   # casefold
    ("product-a",         "product-b",             False),  # genuinely different
]

@pytest.mark.parametrize("left,right,same", NORMALISATION_CASES)
def test_normalisation_equivalence(left, right, same):
    assert (normalise_text(left) == normalise_text(right)) is same

def test_floats_are_rejected():
    with pytest.raises(TypeError):
        dedup_key({"title": "t", "price_minor": 12.99})

def test_key_is_order_independent():
    a = {"title": "t", "sku": "s", "brand": "b"}
    b = {"brand": "b", "sku": "s", "title": "t"}
    assert dedup_key(a)[0] == dedup_key(b)[0]

def test_version_is_stamped():
    _, version = dedup_key({"title": "t"})
    assert version == NORMALISATION_VERSION

Track the collapse rate per source as an ongoing metric. A rise usually means a normalisation change merged more than intended; a fall usually means an identity field stopped being extracted, so every record now looks new. Both are visible within a day and neither is visible from the row count alone.

Compliance & Operational Guardrails #

The normalisation version is stored on every record so a rule change is recognisable.
A rule change is migrated by computing both keys and comparing, never by switching in place.
Merged records retain every contributing source URL and the earliest first-seen timestamp.
Effective deduplication is a data-minimisation control, not merely a storage saving.
Normalisation is applied once, at a single boundary, and shared with change detection and fingerprinting.

Common Mistakes #

Normalising ad hoc per call site. Two fields eventually disagree and the hashes stop matching.
Using NFKC by default. Ligatures, superscripts and full-width characters are rewritten, changing meaning in specifications and identifiers.
Hashing floats. The textual representation varies and the key is not reproducible.
Changing the rule without a version bump. History silently reinterprets and nobody can tell which rule produced a key.
Applying identifier-strength normalisation to prose. Stripping punctuation from a description destroys meaning for no deduplication benefit.

Frequently Asked Questions #

Should normalised text be stored, or only its hash? #

Store the hash for matching and keep the original for display. Storing the normalised form as well is occasionally useful for debugging a suspected over-collapse, but it is a third copy of the same content — reasonable during a migration, wasteful as a permanent arrangement.

How aggressive should normalisation be? #

Enough that presentation differences vanish and no further. The test is whether two texts that normalise identically would be described as the same content by someone reading them. Where that is genuinely arguable, prefer the less aggressive rule: an under-merged dataset has redundant rows, while an over-merged one has lost information that cannot be recovered.

Does this apply to numbers and dates too? #

Yes, and the canonical forms are different: integer minor units for money, ISO 8601 with an explicit zone for timestamps, digits only for identifiers. The principle is identical — one canonical form, applied once, versioned — and the specifics are covered in validating scraped dates and currencies.

Should normalisation strip HTML entities? #

Decode them exactly once, at the parse boundary, before normalisation runs. Decoding again during normalisation is how double-decoding bugs are introduced: &amp; becomes & on the first pass and & on the second, so the same source text hashes differently depending on how many times it happened to pass through. One decode, at one place, recorded in the pipeline’s contract.

How should numeric values inside text be handled? #

Leave them alone in prose and normalise them in identifier fields. A description mentioning “1,299” should keep its formatting, because that text is content; a product code rendered as SKU-1299 and sku 1299 on two pages should collapse. That distinction is exactly what the per-field policy expresses, and attempting a single global rule for both produces either under-merged identifiers or corrupted prose.

Deduplication Strategies for Scraped Data — the ladder this sits at the base of.
Deduplicating Records With Content Hashing — the exact-match layer that depends on this normalisation.
Detecting Character Encoding of Scraped HTML — getting correct text before normalising it.