Data Retention and GDPR Deletion Workflows for Scraped Datasets #

Every record a crawler persists starts a clock. Data retention and GDPR deletion workflows are the engineering discipline that decides how long a scraped record may live, how it expires automatically, and how a specific person’s data is erased on demand — a core responsibility inside the Pipeline Storage, Deduplication & Monitoring section. The problem this solves is concrete: a warehouse full of scraped records that no one ever deletes is not an asset, it is an accumulating liability under GDPR Article 5(1)(e) (storage limitation) and Article 17 (right to erasure). Data engineers, platform owners, and compliance officers all need retention encoded as executable policy rather than a wiki page nobody enforces. This guide covers retention windows, TTL-based expiry, right-to-erasure workflows, soft versus hard deletes, cascading deletes across derived tables, cryptographic shredding, and the deletion audit trail that proves the whole thing works.

Core Principles & Retention Foundation #

Two GDPR articles anchor the design. Article 5(1)(e) says personal data must be kept “no longer than is necessary for the purposes for which it is processed” — the storage-limitation principle that forces a defined retention window on every category of data you hold. Article 17 grants the data subject a right to erasure (“right to be forgotten”): on a valid request, and absent an overriding legal basis, you must delete their personal data “without undue delay,” which regulators read as within one calendar month. These are distinct triggers. Retention expiry is time-based and applies to everyone; erasure is subject-initiated and applies to one person, often ahead of the retention clock.

The correct model is retention as configuration, not as code scattered across cron jobs. Each data category declares a purpose, a lawful basis, a retention window, and a deletion mode. A crawler that also captures personal data should already be governed by the practices in GDPR compliance for scraped personal data; retention is the storage-side continuation of that same lawful-basis reasoning.

Retention Windows as a Policy Table #

Express windows declaratively so they can be reviewed by non-engineers and diffed in version control:

# retention_policy.yaml — the single source of truth for expiry
policies:
  - dataset: product_listings
    contains_personal_data: false
    purpose: price_intelligence
    retention_days: 365
    delete_mode: hard          # non-personal, safe to purge outright
  - dataset: seller_profiles
    contains_personal_data: true
    lawful_basis: legitimate_interest
    purpose: marketplace_analytics
    retention_days: 180
    delete_mode: crypto_shred   # names, contact fields -> unrecoverable
  - dataset: raw_html_snapshots
    contains_personal_data: true
    purpose: debugging_and_reprocessing
    retention_days: 30
    delete_mode: hard
default:
  retention_days: 90
  delete_mode: soft

A field-level contains_personal_data flag matters because it decides which deletion mode is defensible. Non-personal data can be purged on a generous schedule; personal data should carry the shortest window that still serves its stated purpose, honouring data minimisation.

Soft Delete vs Hard Delete vs Cryptographic Shredding #

Soft delete alone never satisfies Article 17 — a row that is merely flagged still exists and is still “processed.” Treat soft delete as a two-phase commit: flag first, then a scheduled job performs the hard delete or crypto shred once a short grace window (24–72 hours) elapses. Cryptographic shredding is the escape hatch for storage you physically cannot edit in place: encrypt each subject’s data under a per-subject key, and to erase them, destroy the key. The ciphertext remains but is permanently undecryptable, which regulators accept as effective erasure.

Implementation Steps #

1. Stamp every record with retention metadata at write time #

The sink is the right place to attach the expiry clock. When records land — whether you are upserting scraped records into Postgres or writing columnar files — compute the expiry from the policy table so downstream jobs never have to guess.

from datetime import datetime, timedelta, timezone
import yaml

with open("retention_policy.yaml") as fh:
    POLICY = {p["dataset"]: p for p in yaml.safe_load(fh)["policies"]}

def with_retention(record: dict, dataset: str) -> dict:
    policy = POLICY[dataset]
    now = datetime.now(timezone.utc)
    record["ingested_at"] = now.isoformat()
    record["expires_at"] = (now + timedelta(days=policy["retention_days"])).isoformat()
    record["delete_mode"] = policy["delete_mode"]
    # Compliance: expiry is derived from declared policy, never hard-coded per call site.
    return record

2. Enforce TTL-based expiry with the store’s native mechanism where possible #

Let the database do the work when it can. Postgres has no built-in row TTL, so a scheduled sweep is standard; Redis, MongoDB, and DynamoDB expire natively.

-- Postgres: index the expiry column so the sweep is a range scan, not a seq scan.
CREATE INDEX IF NOT EXISTS idx_seller_expires ON seller_profiles (expires_at);

-- Phase 1: soft-delete anything past its window (idempotent; safe to re-run).
UPDATE seller_profiles
   SET deleted_at = now()
 WHERE expires_at < now()
   AND deleted_at IS NULL;

# Phase 2: a nightly job promotes soft-deletes to hard deletes past the grace window.
GRACE = timedelta(hours=48)

def purge_expired(conn, dataset: str):
    cutoff = datetime.now(timezone.utc) - GRACE
    with conn.cursor() as cur:
        cur.execute(
            f"DELETE FROM {dataset} WHERE deleted_at < %s RETURNING id",
            (cutoff,),
        )
        purged = [row[0] for row in cur.fetchall()]
    conn.commit()
    for record_id in purged:
        write_deletion_audit(dataset, record_id, reason="ttl_expiry")
    return purged

3. Cascade the delete across every derived copy #

Scraped data rarely lives in one place. A seller profile spawns rows in an analytics table, a dedup index, a search document, and object-storage snapshots. Erasing the primary row while a copy survives in Elasticsearch is still a breach. Declare the fan-out explicitly and let the database enforce referential cascades where it can.

-- Foreign keys with ON DELETE CASCADE make the DB enforce the fan-out for you.
ALTER TABLE seller_analytics
  ADD CONSTRAINT fk_seller
  FOREIGN KEY (seller_id) REFERENCES seller_profiles (id) ON DELETE CASCADE;

# For stores the FK graph does not reach, cascade in application code.
CASCADE_TARGETS = {
    "seller_profiles": [
        lambda sid: es.delete(index="sellers", id=sid, ignore=[404]),
        lambda sid: redis.srem("dedup:seller_hashes", sid),
        lambda sid: s3.delete_object(Bucket="raw-snapshots", Key=f"seller/{sid}.html"),
    ],
}

def cascade_delete(dataset: str, record_id: str):
    for target in CASCADE_TARGETS.get(dataset, []):
        target(record_id)   # Each downstream copy must be reached or erasure is incomplete.

4. Wire the right-to-erasure request path #

An erasure request should resolve one subject to every record they touch, across datasets, and drive them all to the correct deletion mode. Automating this end to end is covered in automating GDPR right-to-erasure deletions; the technique-level skeleton is:

def erase_subject(conn, subject_key: str, request_id: str):
    """Resolve a data subject to all their records and erase per policy."""
    matches = resolve_subject_records(conn, subject_key)  # by email hash, profile URL, etc.
    for dataset, record_id, mode in matches:
        if mode == "crypto_shred":
            destroy_subject_key(record_id)
        else:
            hard_delete(conn, dataset, record_id)
        cascade_delete(dataset, record_id)
        write_deletion_audit(dataset, record_id, reason="article_17_erasure",
                             request_id=request_id)
    # Article 17 requires completion "without undue delay" — target well inside 30 days.

Error Handling & Observability #

A deletion pipeline that fails silently is worse than none, because you will believe data is gone when it is not. Classify failures and keep the sweep idempotent so retries are always safe.

Emit a structured event for every deletion so the audit trail is a byproduct of normal operation:

{
  "timestamp": "2026-07-05T02:14:09Z",
  "event": "record_deleted",
  "dataset": "seller_profiles",
  "record_id": "e3b0c442",
  "delete_mode": "crypto_shred",
  "reason": "article_17_erasure",
  "request_id": "erasure-2026-0417",
  "cascade_targets_cleared": 3,
  "operator": "retention-worker"
}

Track these metrics and alert on the SLOs:

• retention_records_expired_total{dataset} — rate of TTL purges; a sudden zero means the sweep stalled.
• erasure_requests_open — count of erasure requests not yet complete; alert if any exceeds 21 days (buffer before the 30-day legal limit).
• erasure_completion_seconds — histogram of request-to-completion time.
• deletion_cascade_failures_total — must stay at zero; any value is a potential un-erased copy.

Compliance Boundaries #

Retention and deletion are where GDPR moves from paper to proof. The retention window must be justifiable against the stated purpose (Article 5), erasure must be honoured within a month (Article 17), and you must be able to demonstrate both actually happened (Article 5(2), accountability). A CFAA or Terms-of-Service angle exists too: some sites prohibit long-term storage of their content, so a short hard-delete window on raw snapshots doubles as ToS hygiene.

• Every dataset has an entry in retention_policy.yaml Every dataset has an entry in `retention_policy.yaml` with a purpose and lawful basis.
• Personal-data fields use crypto_shred or hard mode, never soft Personal-data fields use `crypto_shred` or `hard` mode, never `soft` alone as the final state.
• The TTL sweep and the erasure path both write a deletion audit record for every row.
• Cascade targets are enumerated for every dataset that has derived copies.
• Backups have a defined rotation such that hard-deleted rows age out, or subjects are crypto-shredded so backup ciphertext is inert.
• Erasure completion time is measured and alerts fire before the 30-day deadline.

On budget maths: if you receive R erasure requests per month and each resolves to an average of N records across S stores, your pipeline must sustain R × N × S delete operations comfortably within the month, with headroom for the largest single subject. Size the worker and connection pool against the p99 subject, not the average.

Common Mistakes #

1. Treating soft delete as erasure. A deleted_at flag hides a row from your UI but the personal data still exists and is still processed — Article 17 is not satisfied. Right: promote to hard delete or crypto shred after a short grace window.
2. Forgetting derived copies. Deleting the Postgres row while the record survives in Elasticsearch, a Parquet file, and a Redis dedup set leaves personal data live. Right: enumerate and cascade to every sink that received a copy.
3. Hard-deleting from immutable storage that cannot be edited. Trying to DELETE a single subject from an append-only Parquet dataset on object storage forces expensive full rewrites. Right: encrypt per subject and crypto-shred by destroying the key.
4. Hard-coding retention windows in cron scripts. Windows buried in code cannot be reviewed by compliance and drift out of sync. Right: keep them in a declarative policy file that both engineers and auditors read.
5. Deleting without an audit record. If you cannot prove a deletion occurred, you cannot demonstrate accountability. Right: write an immutable deletion-audit event for every purge and erasure, keyed by request ID.

Frequently Asked Questions #

How fast must I complete a GDPR erasure request? #

Article 17 requires deletion “without undue delay,” which the GDPR elsewhere defines as within one month of receiving the request. That month can be extended by two further months for complex or numerous requests, but only if you inform the data subject of the extension and the reason within the first month. Engineer the pipeline to complete well inside 30 days — alert at 21 — so the legal window is a safety margin, not your target.

Does deleting a record mean I have to rewrite my backups too? #

Not immediately, but you cannot ignore them. Backups containing personal data are still processing. The pragmatic pattern is a defined backup rotation: hard-deleted rows age out as old backups are overwritten on schedule, and you document that window. For data you cannot afford to wait on, cryptographic shredding is cleaner — once the per-subject key is destroyed, the ciphertext in every backup is permanently unreadable, so no backup rewrite is required.

When is cryptographic shredding preferable to a plain hard delete? #

Use crypto shredding wherever in-place row deletion is expensive or impossible: append-only object storage, columnar Parquet datasets, write-once compliance stores, and backups. Encrypt each subject’s data under a distinct key at write time; erasure becomes a single key-destruction operation instead of a scan-and-rewrite across large immutable files. For a mutable relational table, a plain hard delete is simpler and preferred.

Can I keep any scraped personal data after an erasure request? #

Only where you have an overriding lawful basis that Article 17(3) recognises — for example a legal obligation to retain certain records, or establishment and defence of legal claims. If such a basis applies, you retain only the specific fields it justifies, document the reasoning, and delete everything else. Absent such a basis, the default is full erasure across all stores.

Related guides #

• Pipeline Storage, Deduplication & Monitoring — the section this retention discipline sits within.
• Automating GDPR right-to-erasure deletions — the end-to-end erasure automation this page frames.
• Structured data sinks and warehousing — where retention metadata is stamped and cascades reach.
• GDPR compliance for scraped personal data — the lawful-basis reasoning retention continues on the storage side.
• Deduplication strategies for scraped data — the dedup index is one of the derived copies an erasure must cascade to.