Compliant Web Scraping

Automating GDPR right-to-erasure deletions #

When a data subject exercises their right to erasure under Article 17 of the GDPR, “we deleted it” is not a defensible answer — you must locate every copy of their data across every sink, remove it, and be able to prove you did. For a scraping pipeline that fans records out to Postgres, Parquet on object storage, a search index, and log archives, that is an engineering problem, not a manual one. This page describes an automated erasure workflow: a request queue, cross-sink subject location, hard delete plus tombstone, verification, and an immutable audit log. It is a detail page under Data Retention and GDPR Deletion Workflows, part of the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

Article 17 gives a subject the right to have their personal data erased “without undue delay”, which regulators read as within one month. The obligation is not satisfied by deleting the row a user can see in your app — it extends to every derived copy: warehouse tables, columnar dumps, caches, search indices, and backups. A pipeline that continuously ingests scraped personal data multiplies these copies, so erasure must be systematic and repeatable.

The design tension is between a hard delete (the data is gone) and proof (evidence it was gone, retained for accountability under Article 5(2)). You cannot keep the personal data to prove you deleted it. The resolution is a tombstone: a record keyed by a non-identifying subject reference that says “subject X was erased at time T across sinks Y”, holding no personal data itself, plus an append-only audit log. The tombstone also prevents resurrection — a later re-crawl of the same source must be blocked from re-inserting the erased subject.

Step-by-Step Implementation #

  1. Queue the request. Persist every erasure request with a status and deadline so nothing is lost and SLA breaches are visible.
  2. Resolve the subject to keys. Map the request (an email, a profile URL) to the pseudonymous subject_ref your records are keyed by — never scan raw personal data across sinks.
  3. Fan out hard deletes to each registered sink through a common interface, collecting per-sink row counts.
  4. Write a tombstone so re-crawls cannot resurrect the subject.
  5. Verify that zero records remain in each sink, then close the request.
  6. Append to the audit log — who requested, what was deleted, when, and the verification result.
import hashlib
from datetime import datetime, timezone, timedelta

def subject_ref(identifier: str) -> str:
    # Keyed pseudonym: what records are indexed by, not the raw identifier.
    return hashlib.sha256(("subj:" + identifier.lower().strip()).encode()).hexdigest()

class ErasureWorkflow:
    def __init__(self, sinks: list, audit, tombstones):
        self.sinks = sinks            # each implements delete(ref) + count(ref)
        self.audit = audit
        self.tombstones = tombstones

    def enqueue(self, conn, identifier: str, requested_by: str) -> str:
        ref = subject_ref(identifier)
        deadline = datetime.now(timezone.utc) + timedelta(days=30)  # Art. 12(3)
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO erasure_request (subject_ref, requested_by, status, deadline)"
                " VALUES (%s, %s, 'pending', %s)"
                " ON CONFLICT (subject_ref) DO NOTHING",
                (ref, requested_by, deadline),
            )
        conn.commit()
        return ref

    def execute(self, conn, ref: str) -> None:
        results = {}
        for sink in self.sinks:
            deleted = sink.delete(ref)          # hard delete, returns row count
            results[sink.name] = deleted

        # Tombstone holds NO personal data — only the pseudonymous ref + when.
        self.tombstones.add(ref, erased_at=datetime.now(timezone.utc))

        remaining = {s.name: s.count(ref) for s in self.sinks}
        if any(remaining.values()):
            raise AssertionError(f"erasure incomplete: {remaining}")

        with conn.cursor() as cur:
            cur.execute(
                "UPDATE erasure_request SET status='completed', "
                "completed_at=now() WHERE subject_ref=%s", (ref,))
        conn.commit()
        # Append-only: records the act of deletion without the deleted data.
        self.audit.append(action="erasure", subject_ref=ref,
                          deleted=results, verified=remaining)

Each sink implements the same tiny contract so the workflow stays sink-agnostic. Row stores hard-delete by the pseudonymous key; Parquet cannot mutate a file, so erasure means rewriting the affected partition without the subject’s rows:

class PostgresSink:
    name = "postgres"
    def delete(self, ref: str) -> int:
        with self.conn.cursor() as cur:
            cur.execute("DELETE FROM record WHERE subject_ref = %s", (ref,))
            return cur.rowcount
    def count(self, ref: str) -> int:
        with self.conn.cursor() as cur:
            cur.execute("SELECT count(*) FROM record WHERE subject_ref = %s", (ref,))
            return cur.fetchone()[0]

class ParquetSink:
    name = "parquet"
    def delete(self, ref: str) -> int:
        # Immutable files: read the partition, drop matching rows, atomically
        # rewrite it. Partitioning by a low-cardinality key keeps this cheap.
        return rewrite_partition_excluding(self.dataset, subject_ref=ref)

Because records are keyed by subject_ref rather than raw identifiers, locating a subject is an index lookup in each sink, not a scan of personal data. That keying decision should be made at write time, following GDPR compliance for scraped personal data.

Verification & Testing #

Verification is part of the workflow, not an afterthought — the execute method above already re-counts every sink and refuses to mark the request complete if anything remains. Test the full round trip, including resurrection resistance:

def test_erasure_removes_and_stays_removed(workflow, conn, sinks):
    ref = workflow.enqueue(conn, "[email protected]", requested_by="dpo")
    seed_records(sinks, ref, n=5)

    workflow.execute(conn, ref)
    assert all(s.count(ref) == 0 for s in sinks)          # gone everywhere

    # A later re-crawl must NOT resurrect the subject.
    ingest_scraped_record(sinks, subject_ref=ref, tombstones=workflow.tombstones)
    assert all(s.count(ref) == 0 for s in sinks)          # still gone

For an operational check, query the request queue for anything past its deadline: SELECT subject_ref, deadline FROM erasure_request WHERE status='pending' AND deadline < now() should always return zero rows. Wire that into a Prometheus alert so a stalled erasure is visible before it becomes a regulatory breach.

Compliance & Operational Guardrails #

  • The tombstone must contain no personal data. It exists to prove erasure and to block re-ingestion; storing the erased identifier in it would recreate the very data you deleted. Key it on the pseudonymous subject_ref only.
  • Address backups explicitly. Regulators accept that erasing from immutable backups on a rolling schedule is proportionate — document the backup rotation window and re-apply the tombstone if a restore reintroduces the subject.
  • Keep the audit log append-only and access-controlled. Article 5(2) accountability requires evidence you can produce on demand; a mutable log undermines it.
  • Honour the one-month deadline. Track deadline per request and alert before it lapses; extensions under Article 12(3) must themselves be logged.

Common Mistakes #

  1. Soft-deleting when erasure is required. A deleted=true flag leaves the personal data intact and does not satisfy Article 17. Hard-delete the data; keep only a non-identifying tombstone.
  2. Forgetting derived sinks. Deleting from Postgres but leaving copies in Parquet, a search index, or logs means the data is not erased. Register every sink behind one interface.
  3. No resurrection guard. Without a tombstone, the next crawl re-inserts the subject and silently undoes the erasure.

Frequently Asked Questions #

Do I have to delete the subject’s data from backups too? #

Yes, the right to erasure covers backups, but regulators accept a proportionate approach: if backups are immutable and rotate on a defined schedule, you may let the erased data expire with the normal rotation rather than restoring and rewriting every archive. Document the rotation window, and if you restore a backup that predates an erasure, re-run the tombstone check so the subject is removed again before the data returns to service.

How does a tombstone let me prove deletion without keeping the personal data? #

The tombstone stores only a keyed pseudonym (subject_ref), the erasure timestamp, and which sinks were cleared — none of which is personal data on its own. Combined with an append-only audit log, it demonstrates that a specific request was fulfilled and when, satisfying the Article 5(2) accountability duty, while the underlying name, email, or address is genuinely gone.