Compliant Web Scraping

GDPR Compliance for Scraped Personal Data #

The moment a crawler ingests a name, an email address, a profile URL, or even an IP tied to a natural person, it crosses from “collecting documents” into “processing personal data” under the General Data Protection Regulation — and inherits obligations that no amount of clever request throttling will discharge. This technique page, part of the Compliance & Ethical Crawling Foundations section, walks through how to decide when scraped data is regulated, how to establish a lawful basis before the first request, and how to bake minimisation, transparency, and data-subject rights into the pipeline rather than bolting them on after an audit. It is written for engineers who own the ingest path and the compliance officers who have to defend it.

Personal-data lawful-basis flowPersonal-data lawful-basis flowPersonal-data lawful-basis flow1DetectPII2Lawfulbasis?3Minimize& tokenize4Log &retainPersonal data is only kept when a lawful basis and minimisation are satisfied.
Personal data is only kept when a lawful basis and minimisation are satisfied.

Core Principles & Regulatory Foundation #

GDPR bites on processing of personal data about an identified or identifiable natural person (a data subject). Scraping is processing. Storing is processing. Deleting is processing. The threshold for “personal data” under Article 4(1) is deliberately low: anything that can single out an individual, directly or in combination with other data you hold, qualifies. A publicly visible email on a company contacts page is still personal data. So is a pseudonymous forum handle if you can link it back to a person.

Publication does not neuter the regulation. The Weltimmo and Ryneš line of CJEU cases, and the regulators’ consistent guidance, make clear that “the data was public” is not a lawful basis and not an exemption. You still need a legal ground under Article 6, transparency under Articles 13–14, and you still owe data subjects their rights.

When scraped data is “personal data” #

Use this table to classify fields at the point of extraction, not after storage. The classification drives every downstream control.

Scraped field Personal data? Special category (Art. 9)? Notes
Full name, job title Yes No Directly identifying
Business email [email protected] Yes No Still personal even if “professional”
Generic email [email protected] Usually no No No natural person singled out
Profile / avatar URL Yes Maybe Image may reveal ethnicity, health
Forum post text Yes Maybe May reveal politics, religion, health, sexuality
IP address, device fingerprint Yes No Breyer — identifiable via third party
Product SKU, price No No Not about a natural person
Trade-union membership, biometric Yes Yes Art. 9 near-prohibition applies

Article 9 special-category data (health, ethnicity, political opinion, religion, sexual orientation, biometric, trade-union membership) is effectively prohibited to process unless a narrow exception applies. Legitimate interest is not one of those exceptions. If your crawl target is a health forum or a political community, assume you cannot scrape identifiable member data at all without explicit consent, which is impractical to obtain — so exclude those sources.

Consent under GDPR must be freely given, specific, informed, and unambiguous. You cannot obtain valid consent from someone whose page you scraped without their knowledge, so consent is almost never the workable basis for scraping. That leaves legitimate interest under Article 6(1)(f) as the realistic ground for most compliant scraping of business or public-professional data.

Legitimate interest is not a free pass. It requires a documented three-part Legitimate Interests Assessment (LIA):

  1. Purpose test — is there a genuine, lawful, specifically articulated interest? (“Building a B2B sales-lead dataset” is specific; “collecting data that might be useful” is not.)
  2. Necessity test — is scraping necessary, or could you achieve the purpose with less intrusive means (a licensed dataset, an API, aggregate data)?
  3. Balancing test — do the individual’s rights and reasonable expectations override your interest? Someone posting a work email expects business contact; they do not expect enrolment in an automated profiling system.

Record the LIA outcome as a versioned artifact tied to the crawl configuration, and make the balancing test fail closed for anything touching Article 9 data or children’s data.

Implementation Steps #

The controls below turn the legal analysis into pipeline code. Classify, minimise, and log at ingest — before personal data is ever persisted in a warehouse.

Step 1 — Classify and tag fields at extraction #

Attach a data-classification label to every extracted field so downstream stages (storage, retention, erasure) can act on it without re-parsing raw HTML.

from dataclasses import dataclass, field
from enum import Enum

class DataClass(Enum):
    NON_PERSONAL = "non_personal"
    PERSONAL = "personal"          # Art. 4 personal data
    SPECIAL = "special_category"   # Art. 9 — near-prohibited

@dataclass
class ExtractedField:
    name: str
    value: str
    data_class: DataClass
    lawful_basis: str | None = None  # e.g. "legitimate_interest:lia-2026-014"

SPECIAL_KEYWORDS = {"health", "diagnosis", "religion", "union", "orientation"}

def classify(name: str, value: str) -> DataClass:
    lname = name.lower()
    if any(k in lname for k in ("email", "name", "phone", "profile", "ip")):
        # Generic role addresses are not about a natural person
        if lname == "email" and value.split("@")[0] in {"info", "contact", "sales"}:
            return DataClass.NON_PERSONAL
        return DataClass.PERSONAL
    if any(k in lname for k in SPECIAL_KEYWORDS):
        return DataClass.SPECIAL
    return DataClass.NON_PERSONAL

If classify returns DataClass.SPECIAL, the correct behaviour is to drop the field (or the whole record) and emit a compliance event — do not persist it pending review.

Step 2 — Minimise and pseudonymise before persistence #

Article 5(1)© (data minimisation) means you store only fields the documented purpose actually needs. Article 32 encourages pseudonymisation as a security-of-processing measure. Do both at the ingest boundary so raw identifiers never hit durable storage in cleartext where they are not required.

import hashlib
import hmac

# Pepper stored in a secrets manager, NOT in code or the DB.
_PEPPER = load_secret("PSEUDONYM_PEPPER")

def pseudonymise(value: str) -> str:
    # Keyed hash so the mapping is not reversible by rainbow tables,
    # and can be rotated/retired to effect erasure.
    return hmac.new(_PEPPER, value.encode(), hashlib.sha256).hexdigest()

def minimise(record: dict, needed: set[str]) -> dict:
    return {k: v for k, v in record.items() if k in needed}

Keep the pseudonymisation key separate from the pseudonymised data. Destroying the key is itself a valid route to rendering data effectively anonymous, which connects directly to your deletion strategy in the data retention and GDPR deletion workflows technique.

Step 3 — Satisfy Article 14 transparency (data not obtained from the subject) #

Because you collect from a third-party site rather than from the person, Article 14 applies: you must proactively inform data subjects — normally within one month — of your identity, purposes, legal basis (your legitimate interest), the categories of data, recipients, retention period, and their rights. There is a narrow “disproportionate effort” exemption under Art. 14(5)(b), but regulators (see the Polish DPA’s Bisnode fine) expect you to at least publish a prominent privacy notice and, where you hold contact details, to notify directly.

def article14_notice_targets(records: list[dict]) -> list[str]:
    # Where we hold a deliverable contact address, direct notice is expected.
    return [r["email"] for r in records
            if r.get("email") and r.get("data_class") == "personal"]

Maintain a public, versioned privacy notice URL and record which notice version applied to each ingest batch, so you can prove what a data subject was told and when.

Step 4 — Run a DPIA when profiling or scale demands it #

A Data Protection Impact Assessment (Article 35) is mandatory when processing is likely to result in high risk — which explicitly includes systematic evaluation/profiling and large-scale processing. Bulk scraping of personal data to build profiles is a textbook DPIA trigger. Treat the DPIA as a gated CI artifact: no new personal-data source ships without one.

Error Handling & Observability #

Compliance failures are silent by default — a mis-classified special-category field persists happily until a regulator asks. Make the pipeline loud about the states that matter, and keep an immutable audit trail. Route these events into the structured logging and log shipping pipeline so the audit record lives outside the application database and survives its deletion.

Condition Class Action
DataClass.SPECIAL field extracted Terminal Drop field, emit gdpr.special_category.dropped, alert
No lawful_basis on a personal field Terminal Reject record, block persistence
Article 14 notice version missing Terminal Fail batch — cannot prove transparency
Pseudonymisation key unavailable Retriable Buffer, do not write cleartext
Erasure request received Terminal (SLA) Trigger deletion workflow within 30 days 
{
  "event": "gdpr.ingest.record",
  "trace_id": "b8f2...",
  "source_url": "https://example.com/team",
  "fields_personal": 3,
  "fields_special_dropped": 1,
  "lawful_basis": "legitimate_interest:lia-2026-014",
  "notice_version": "privacy-2026-06",
  "retention_expires": "2027-01-05T00:00:00Z"
}

Useful Prometheus signals: gdpr_special_category_dropped_total (alert on any non-zero rate — it means you are crawling the wrong source), gdpr_records_without_basis_total (alert > 0), and gdpr_erasure_pending_seconds (alert when the oldest open request approaches the 30-day SLA).

Compliance Boundaries #

GDPR does not stand alone: the same crawl may implicate the ePrivacy rules on cookies, national implementations with stricter carve-outs, and — for the CFAA and contractual angle — the site’s own terms, which is why lawful-basis work sits alongside documenting scraping authorization and data lineage. The engineering guardrails:

A quick scale check: if a source yields 50,000 personal records and your Article 14 obligation is direct notice where you hold an email, budget for the send volume and the objection-handling load before you run the crawl — an unservable rights backlog is itself a violation.

Common Mistakes #

  1. “It was public, so GDPR doesn’t apply.” Wrong. Public availability changes the balancing test slightly; it never removes the need for a lawful basis, transparency, or rights handling.
  2. Defaulting to consent as the lawful basis. You cannot get valid consent from someone you scraped silently. Use — and document — legitimate interest instead, and accept that some sources fail the balancing test.
  3. Skipping Article 14 because “notifying everyone is too hard.” The disproportionate-effort exemption is narrow and was tested against a real fine. At minimum publish a versioned notice and directly notify where you hold contact details.
  4. Storing raw identifiers indefinitely. No retention limit breaches Art. 5(1)(e). Set a TTL at ingest and enforce it with an automated deletion job.
  5. Treating pseudonymisation as anonymisation. Keyed-hash data is still personal data if you (or anyone) can reverse or re-link it. Only irreversible severing — e.g. destroying the key — approaches anonymity.

Frequently Asked Questions #

It can be, but “public” is not itself a lawful basis. You still need a valid Article 6 ground — in practice a documented legitimate interest that passes a three-part balancing test — plus Article 14 transparency and working data-subject rights. If the balancing test favours the individual, or the data is special-category, the answer is no.

Do I have to tell people I scraped their data? #

Yes, in most cases. Article 14 governs data not collected from the subject and requires you to provide your identity, purposes, legal basis, retention, and their rights, normally within a month. There is a narrow disproportionate-effort exemption, but regulators expect a prominent published privacy notice and direct notification where you hold usable contact details.

Can legitimate interest cover scraping special-category data like health or political views? #

No. Article 9 special-category data is near-prohibited and legitimate interest is not one of the listed exceptions. The only realistic ground is explicit consent, which you cannot obtain by silent scraping. Exclude such sources at the source level rather than filtering after the fact.

How does pseudonymisation help my GDPR posture if the data is still personal? #

Pseudonymisation is a security-of-processing measure under Article 32 that reduces risk and can strengthen your legitimate-interest balancing, but pseudonymised data remains personal data because it is re-linkable. Its real leverage comes later: destroying the keyed-hash pepper is a clean, verifiable way to render whole cohorts of records effectively irreversible, supporting bulk erasure and retention enforcement.

When is a DPIA mandatory for a scraping project? #

Whenever processing is likely to be high-risk — which Article 35 and the regulators tie explicitly to large-scale processing and systematic profiling. Bulk collection of personal data to build or enrich profiles hits both criteria, so treat the DPIA as a required, version-controlled gate that must be signed off before a new personal-data source goes live.