Compliant Web Scraping

Documenting Scraping Authorization and Data Lineage #

When a data-collection practice is challenged — by a site owner, a regulator, or opposing counsel — the question is rarely “did you write good code?” It is “can you prove what you were permitted to collect, and can you trace every record back to where and when it came from?” Answering both requires two artifacts your pipeline must produce as a byproduct of running: an authorization record (what let you scrape) and a lineage record (the provenance of each datum). This page, a detail under the Mapping Terms of Service for Scrapers technique in the Compliance & Ethical Crawling Foundations section, shows how to capture both for legal defensibility and audits.

Problem Framing #

Authorization and lineage answer different questions and fail in different ways. Authorization is the basis — a signed data-sharing agreement, an API key issued under documented terms, a recorded acceptance of a ToS, or a documented legitimate interest. Lineage is the chain of custody — for a given field in your warehouse, which URL produced it, at what timestamp, under which robots.txt decision, and through which transforms.

The failure modes are asymmetric. Missing authorization is an existential legal problem: without it, a scrape can be characterised as exceeding authorised access, which is exactly the terrain the Understanding CFAA implications for web scraping page covers. Missing lineage is an evidential problem: even if you were authorised, you cannot prove a specific record was collected within scope, cannot honour a deletion request scoped to a source, and cannot reproduce a dataset for audit. Both need to be captured at collection time, because reconstructing them months later from raw dumps is unreliable and self-serving.

Step-by-Step Implementation #

  1. Create an immutable authorization record per source before the first request, capturing the basis, its evidence, and its validity window.
  2. Snapshot the terms you relied on — hash the ToS/API-terms text so you can prove which version was in force when you collected.
  3. Stamp every fetched response with a lineage envelope: source URL, canonical URL, fetch timestamp, HTTP status, the robots.txt allow/deny decision, and the authorization ID that covered it.
  4. Append a transform trail as the record moves through parsing, normalisation, and enrichment, so each mutation is attributable.
  5. Persist lineage to append-only storage separate from the mutable data warehouse, and ship it alongside your operational logs.
import hashlib
import json
import uuid
from datetime import datetime, timezone

def _now() -> str:
    return datetime.now(timezone.utc).isoformat()

def authorization_record(source: str, basis: str, terms_text: str,
                         valid_until: str, evidence_uri: str) -> dict:
    """One immutable row per source describing WHY we may scrape it."""
    return {
        "auth_id": f"auth-{uuid.uuid4().hex[:12]}",
        "source": source,
        "basis": basis,  # e.g. "api_terms" | "written_permission" | "legitimate_interest"
        "terms_sha256": hashlib.sha256(terms_text.encode()).hexdigest(),
        "evidence_uri": evidence_uri,  # signed PDF, email thread, contract in object store
        "recorded_at": _now(),
        "valid_until": valid_until,
    }

def lineage_envelope(response, canonical_url: str, robots_decision: str,
                     auth_id: str) -> dict:
    """Provenance stamped onto every fetched resource."""
    body = response.content
    return {
        "lineage_id": f"lin-{uuid.uuid4().hex[:16]}",
        "source_url": response.url,
        "canonical_url": canonical_url,
        "fetched_at": _now(),
        "http_status": response.status_code,
        "content_sha256": hashlib.sha256(body).hexdigest(),
        "robots_decision": robots_decision,  # "allow" | "disallow" | "no_robots"
        "auth_id": auth_id,                   # ties this fetch to its authorization
        "transforms": [],                     # appended as the record is processed
    }

def record_transform(envelope: dict, stage: str, detail: str) -> dict:
    envelope["transforms"].append({"stage": stage, "detail": detail, "at": _now()})
    return envelope

A completed lineage envelope for one record is a self-contained audit exhibit:

{
  "lineage_id": "lin-9f2c4a1b7e0d3c88",
  "source_url": "https://example.com/directory/jane-doe",
  "canonical_url": "https://example.com/directory/jane-doe",
  "fetched_at": "2026-07-05T09:14:22+00:00",
  "http_status": 200,
  "content_sha256": "d2f7...",
  "robots_decision": "allow",
  "auth_id": "auth-3b1f9c0a2d55",
  "transforms": [
    {"stage": "parse", "detail": "extracted name,email via css .contact", "at": "2026-07-05T09:14:23+00:00"},
    {"stage": "normalise", "detail": "email lowercased, pseudonymised", "at": "2026-07-05T09:14:23+00:00"}
  ]
}

Verification & Testing #

Prove that (a) no record can be persisted without a valid authorization, and (b) lineage round-trips faithfully.

def test_every_record_carries_a_valid_auth():
    auth = authorization_record(
        "example.com", "api_terms", "TERMS v3 ...",
        valid_until="2027-01-01T00:00:00Z", evidence_uri="s3://legal/example-terms-v3.pdf")
    env = lineage_envelope(_FakeResp("https://example.com/x", 200, b"<html>"),
                           "https://example.com/x", "allow", auth["auth_id"])
    assert env["auth_id"] == auth["auth_id"]
    # A fetch with no matching authorization must be rejected upstream.
    assert env["robots_decision"] in {"allow", "disallow", "no_robots"}

def test_terms_hash_is_stable_and_detects_change():
    a = authorization_record("s", "api_terms", "TERMS A", "2027-01-01T00:00:00Z", "uri")
    b = authorization_record("s", "api_terms", "TERMS A (edited)", "2027-01-01T00:00:00Z", "uri")
    assert a["terms_sha256"] != b["terms_sha256"]  # any wording change is provable

Operationally, a lineage store should let you answer, in one query, “show every record sourced from example.com between two dates whose robots_decision was disallow” — a non-empty result is an incident, and the ability to run the query at all is the audit capability you are building.

Compliance & Operational Guardrails #

  • Authorization must predate collection. A record created after a dispute begins is worth little. Timestamp it, store the underlying evidence (signed agreement, API key issuance, ToS acceptance) in immutable object storage, and reference it by auth_id.
  • Hash the terms you relied on. ToS pages change silently; a terms_sha256 lets you prove which version governed a given fetch, complementing the versioned-snapshot approach in the parent ToS mapping technique.
  • Keep lineage append-only and separate. Ship it to the structured logging and log shipping pipeline so provenance survives even if the warehouse row is later deleted for a right-to-erasure request — you retain proof of what you did without retaining the personal data itself.
  • Bind lineage to legal exposure. The robots_decision and auth_id fields are precisely the facts that determine whether access was authorised, which is why they belong in the same envelope rather than in scattered logs.

Common Mistakes #

  1. Relying on ambient logs as lineage. Web-server access logs record that a request happened, not why it was authorised or how the resulting data was transformed. Lineage must be a first-class, structured artifact attached to the record.
  2. Capturing authorization once and never revalidating. A valid_until that has passed, or a ToS hash that no longer matches the live page, means your basis has lapsed. Re-check on a schedule and block sources whose authorization has expired.
  3. Storing lineage in the same mutable table as the data. When you delete a record for erasure or dedup, you erase its provenance too. Keep the audit trail in append-only storage so you can prove past conduct without hoarding the underlying personal data.

Frequently Asked Questions #

What counts as adequate authorization to scrape a site? #

It depends on the source, but strong forms include a signed data-sharing or licensing agreement, an API key issued under documented terms you accepted, or a recorded acceptance of a clickwrap ToS that permits your use. A documented legitimate-interest assessment can support collection of public data. Weak or absent authorization — scraping in the face of a Disallow or an explicit prohibition — is what turns a technical activity into a legal exposure, so record the basis and its evidence before you start.

Why hash the terms of service instead of just saving a copy? #

Do both: store the copy and record its SHA-256. The hash gives you a compact, tamper-evident fingerprint that proves which exact wording was live when you collected, so if the publisher later edits the ToS you can show what you actually agreed to. Comparing a fresh hash against the stored one is also how you detect that the terms changed and your authorization needs re-review.

How does data lineage help with a GDPR erasure request? #

Lineage lets you locate every record derived from a given source or subject and the transforms applied to each, so you can delete precisely the right rows and prove you did. Because the lineage envelope is stored append-only and separately from the data, you can retain evidence that you handled the request correctly — the lineage_id, timestamps, and decision — without retaining the personal data you were asked to erase.