Pseudonymising Scraped Personal Data at Ingest #

Some pipelines genuinely need to distinguish one person’s records from another’s without needing to know who that person is — deduplication, session reconstruction, longitudinal analysis. Pseudonymisation serves exactly that case: replace the identifier with a stable, non-reversible token at the moment of ingest, and the dataset keeps its analytical value while holding far less. This guide implements it, as part of GDPR compliance for scraped personal data in the Compliance & Ethical Crawling Foundations section.

Problem Framing #

Pseudonymisation is often described as if it were anonymisation. It is not, and the distinction has practical consequences. Pseudonymised data is still personal data: the link to an individual has been made harder to follow, not removed, and every obligation — lawful basis, retention, erasure, transparency — continues to apply. What it does buy is a substantial reduction in risk if the dataset leaks, and a clean separation between the analytical pipeline and the identifying values.

Pseudonymised data is personal data with less exposure, not anonymous data.

Two properties define a usable pseudonym. It must be stable: the same input must always produce the same token, or records about one person will not group. And it must be non-reversible without a secret: a plain hash of an email address is trivially reversed by hashing a dictionary of addresses, so the construction has to be keyed.

The third property, often forgotten, is that the pseudonym must be queryable by the erasure workflow. A token nobody can compute from a subject’s identifier is a token that makes erasure impossible, which turns a privacy control into a privacy failure.

Step-by-Step Implementation #

1. Normalise, then key the hash #

Removing the raw value is what makes the control real rather than decorative.

import hashlib
import hmac
import unicodedata

def normalise_identifier(value: str, kind: str) -> str:
    """Canonical form so trivial variations produce the same pseudonym."""
    text = unicodedata.normalize("NFKC", value).strip().lower()
    if kind == "email":
        local, _, domain = text.partition("@")
        local = local.split("+", 1)[0]              # subaddressing is the same mailbox
        return f"{local}@{domain}"
    if kind == "phone":
        return "".join(ch for ch in text if ch.isdigit())
    return " ".join(text.split())

def pseudonym(value: str, kind: str, key: bytes) -> str:
    """Stable, keyed, non-reversible token. `key` lives in the secret store."""
    canonical = f"{kind}:{normalise_identifier(value, kind)}"
    return hmac.new(key, canonical.encode("utf-8"), hashlib.sha256).hexdigest()

Including the kind in the hashed material prevents a phone number and an account identifier that happen to share digits from colliding into one pseudonym. Normalisation decides how aggressively records group: stripping subaddressing merges [email protected] with [email protected], which is usually correct for an email but is a judgement worth recording rather than leaving implicit in code.

2. Keep the key out of the pipeline’s reach #

The key is what makes the pseudonym non-reversible, and a key stored beside the data provides no protection at all. Load it from a secret store at start-up, never log it, and never write it to the same system that holds the pseudonymised records.

from functools import lru_cache

@lru_cache(maxsize=1)
def pseudonym_key() -> bytes:
    key = secrets.get("pseudonym/hmac-key")          # secret store, not env, not config file
    if len(key) < 32:
        raise RuntimeError("pseudonym key must be at least 32 bytes")
    return key

Rotating the key invalidates every existing pseudonym, so treat rotation as a migration: compute both the old and the new token for a transition period, migrate downstream joins, then drop the old column. Rotating casually is how a dataset silently stops grouping records that belong together.

3. Apply it at the boundary, before anything is stored #

Pseudonymisation is only meaningful if the raw value never reaches durable storage. That means applying it in the extraction stage, not in a nightly job over an already-written table.

from dataclasses import dataclass

PSEUDONYMISE = {
    "author_email": "email",
    "seller_phone": "phone",
    "reviewer_handle": "handle",
}

@dataclass
class IngestResult:
    record: dict
    pseudonymised_fields: list[str]

def apply_pseudonymisation(record: dict) -> IngestResult:
    key, applied = pseudonym_key(), []
    for field, kind in PSEUDONYMISE.items():
        raw = record.pop(field, None)                # pop: the raw value does not continue
        if raw:
            record[f"{field}_pseudonym"] = pseudonym(raw, kind, key)
            applied.append(field)
    record["pseudonymisation_version"] = "v2"
    return IngestResult(record, applied)

pop rather than a read is the operative detail. Leaving the original field in the record and adding a pseudonym beside it achieves nothing, and it is the single most common way this control is implemented ineffectively.

Note also that raw responses in the landing zone still contain the original values. That is acceptable only if the landing zone has a short, enforced retention and restricted access — the arrangement described in the storage section.

4. Maintain the erasure index in the same transaction #

The pseudonym is what the erasure workflow will search for, so the mapping has to exist before a request arrives.

def write_with_index(conn, record: dict, applied: list[str]) -> None:
    with conn.transaction():
        record_id = insert_record(conn, record)
        for field in applied:
            conn.execute(
                """INSERT INTO subject_index (subject_key, store_name, record_id)
                   VALUES (%s, %s, %s) ON CONFLICT DO NOTHING""",
                (bytes.fromhex(record[f"{field}_pseudonym"]), "listings_curated", record_id),
            )

Because the pseudonym is computed deterministically from the identifier, an erasure request naming an email address can be resolved by recomputing the token and querying the index — no lookup table of raw values is needed anywhere, which is exactly the point. The full workflow is in automating GDPR right-to-erasure deletions.

Verification & Testing #

A token nobody can recompute makes erasure impossible rather than easier.

def test_pseudonym_is_stable_and_normalised():
    key = b"x" * 32
    a = pseudonym(" [email protected] ", "email", key)
    b = pseudonym("[email protected]", "email", key)
    assert a == b

def test_pseudonym_depends_on_the_key():
    assert pseudonym("[email protected]", "email", b"x" * 32) != pseudonym("[email protected]", "email", b"y" * 32)

def test_raw_value_never_survives_ingest():
    result = apply_pseudonymisation({"author_email": "[email protected]", "title": "t"})
    serialised = json.dumps(result.record)
    assert "[email protected]" not in serialised
    assert "author_email" not in result.record

def test_kind_prevents_cross_type_collision():
    key = b"x" * 32
    assert pseudonym("447700900123", "phone", key) != pseudonym("447700900123", "handle", key)

The third test belongs in the suite permanently. It is the assertion that fails when someone adds a field to the extractor and forgets the pseudonymisation map, which is the realistic way raw identifiers re-enter a dataset.

Compliance & Operational Guardrails #

Pseudonymised data remains personal data; retention, lawful basis and erasure obligations are unchanged.
The key lives in a secret store, is at least 32 bytes, and is never written beside the data.
Raw identifiers are removed from the record, not merely supplemented with a token.
The subject index is written in the same transaction as the record.
The normalisation rules and the pseudonymisation version are recorded on every record.
Key rotation is planned as a migration, never performed casually.

Common Mistakes #

Using an unkeyed hash. A dictionary attack recovers the identifiers in minutes, so the control provides essentially no protection.
Keeping the raw value “just in case”. It defeats the entire exercise and is the state auditors find most often.
Pseudonymising in a nightly batch. The raw values sat in durable storage in the meantime, and in backups taken during that window.

Frequently Asked Questions #

Does pseudonymisation remove the need for a lawful basis? #

No. It reduces risk and demonstrates data minimisation, both of which strengthen an assessment, but the data remains personal data and still requires a basis, a retention period and a route to erasure. Where a basis rests on legitimate interests, pseudonymisation is a meaningful factor in the balancing — see recording a legitimate interest assessment.

When is anonymisation possible instead? #

Only when no reasonable means could re-identify anyone, which usually requires aggregation and the removal of the record-level grain entirely. A per-record token is never anonymisation, however strong the hash: the ability to single out an individual’s records is precisely what identifiability means in this context.

How should the normalisation rules be chosen? #

By deciding what “the same person” means for your use case, and writing it down. Merging email subaddresses is right for deduplicating an author but wrong if the subaddress is meaningful in your domain. Record the rules and their version on the record so a later change is recognisable rather than silently regrouping the dataset.

Can a pseudonym be shared with a downstream consumer? #

Only if that consumer is inside the same controller boundary and has no access to the key. A token shared alongside the key is a plain identifier, and a token shared with a party who holds an overlapping dataset may be re-identifiable by joining on other fields — pseudonymity is a property of the whole context, not of the token in isolation. Where an external consumer genuinely needs to group records by subject, derive a second, consumer-specific token from a different key so that two consumers cannot join their datasets on it. That costs one more HMAC per record and removes an entire class of downstream re-identification.

How does pseudonymisation interact with deduplication? #

Well, because a stable token is exactly what deduplication needs: two records about the same person collapse on the pseudonym without either record holding the identifier. The one constraint is that the normalisation used to build the pseudonym must match the normalisation used elsewhere, or the same person produces two tokens. Sharing the normalisation function between the two paths, rather than reimplementing it, is what keeps them consistent.

GDPR Compliance for Scraped Personal Data — the obligations this control supports but does not remove.
Recording a Legitimate Interest Assessment — documenting the basis that permits the collection.
Automating GDPR Right-to-Erasure Deletions — resolving a subject to records through the index this guide writes.