Recording a Legitimate Interest Assessment #

Most web scraping of personal data that is not consent-based rests on legitimate interests, and that basis is only available if the balancing test has actually been performed and written down. This guide covers what the assessment has to contain, how to keep it as a versioned artefact next to the code rather than a document nobody can find, and how to wire its outcome into the crawler’s configuration. It belongs to GDPR compliance for scraped personal data in the Compliance & Ethical Crawling Foundations section.

Problem Framing #

Engineers reasonably ask why an assessment is an engineering concern at all. Three reasons make it one. The outcome of the assessment determines the crawler’s configuration — which fields may be collected, from which sources, at what retention. The assessment has to be produced on request, which means it needs to be findable and version-matched to what the pipeline was doing at the time. And the mitigations it relies on — minimisation, pseudonymisation, short retention — are controls somebody has to implement and keep working, or the assessment describes a system that does not exist.

Skipping the fourth step produces an assessment describing a system that does not exist.

The assessment is a three-part test: is there a genuine interest, is the processing necessary to achieve it, and does the individual’s interest override yours. Skipping any part produces a document that reads as a justification rather than an assessment, which is worse than none.

Step-by-Step Implementation #

1. Keep it as a structured artefact in the repository #

The mitigations block generates the registry entry, so the two cannot diverge.

A prose document in a shared drive drifts from reality within a quarter. A structured file that lives beside the crawler configuration is reviewed in the same pull request as the change it describes.

# assessments/LIA-2026-041.yaml
id: LIA-2026-041
version: 2
status: approved
approved_by: data-governance
approved_on: 2026-06-14
review_due: 2026-12-14
supersedes: LIA-2026-018

purpose:
  summary: >
    Monitor publicly listed prices and availability across trade marketplaces to
    produce aggregated market-rate benchmarks for subscribers.
  interest: commercial
  third_party_benefit: >
    Subscribers gain comparable pricing data; sellers benefit from a transparent
    market rate. No individual-level output is published.

necessity:
  alternatives_considered:
    - option: licensed data feed
      rejected_because: covers 11% of the marketplaces in scope
    - option: aggregate-only public reports
      rejected_because: published quarterly; benchmarks require weekly granularity
  minimal_data_argument: >
    Only seller display name and listing attributes are collected. Contact details,
    addresses and review text are explicitly excluded by the extraction schema.

personal_data:
  categories: [seller_display_name, seller_account_id]
  special_categories: none
  sources: [public marketplace listing pages]
  volume_estimate_records_per_month: 2400000

balancing:
  reasonable_expectation: >
    Listings are published to a public marketplace to attract buyers; commercial
    aggregation of listing attributes is within the expectation of a trade seller.
  potential_impact: >
    Low. No profiling of individuals, no automated decisions, no contact, no
    publication of individual-level records.
  mitigations:
    - identifiers pseudonymised at ingest (see pseudonymisation control)
    - retention limited to 180 days for record-level data
    - published processing notice with an objection route
    - no combination with data from other sources at individual level
  outcome: proceed

controls:
  retention_days: 180
  pseudonymise_fields: [seller_account_id]
  excluded_fields: [seller_email, seller_phone, seller_address, review_text]
  objection_route: https://example.org/crawler#objections

The controls block is the part that makes the assessment operational: it is the same vocabulary the crawl registry uses, so the configuration can be generated from the assessment rather than written independently and hoped to match.

2. Enforce the controls from the assessment #

import yaml
from pathlib import Path

class AssessmentError(RuntimeError):
    """Raised when configuration and assessment disagree. Never caught by workers."""

def load_assessment(ref: str) -> dict:
    data = yaml.safe_load(Path(f"assessments/{ref}.yaml").read_text())
    if data["status"] != "approved":
        raise AssessmentError(f"{ref} is {data['status']}, not approved")
    return data

def assert_registry_matches(entry, assessment: dict) -> None:
    controls = assessment["controls"]
    if entry.retention_days > controls["retention_days"]:
        raise AssessmentError(
            f"{entry.host}: retention {entry.retention_days}d exceeds "
            f"assessed {controls['retention_days']}d ({assessment['id']})")
    overreach = set(entry.extracted_fields) & set(controls["excluded_fields"])
    if overreach:
        raise AssessmentError(
            f"{entry.host}: extracts fields excluded by {assessment['id']}: {sorted(overreach)}")
    missing = set(controls["pseudonymise_fields"]) - set(entry.pseudonymised_fields)
    if missing:
        raise AssessmentError(
            f"{entry.host}: fields require pseudonymisation per {assessment['id']}: {sorted(missing)}")

Run this in continuous integration and again in the crawler’s start-up gate. The failure mode it prevents is the common one: an extractor gains a field months after the assessment was written, nobody re-reads the assessment, and the pipeline is now collecting data the balancing test never considered.

3. Treat an overdue review as an expired basis #

from datetime import date

def assert_current(assessment: dict) -> None:
    due = date.fromisoformat(str(assessment["review_due"]))
    if due < date.today():
        raise AssessmentError(
            f"{assessment['id']} review was due {due}; the basis is not current")

This mirrors the registry’s expiry rule, and for the same reason: circumstances change. A source that was plainly public becomes gated, a volume that was modest grows tenfold, a purpose that was internal becomes a published product. Each of those changes the balancing, and a calendar-driven re-read is the only mechanism that reliably catches them.

4. Make objections reachable and actioned #

The balancing almost always relies on the individual’s ability to object, which means the route has to work. Publish it on the same page your crawler token points at, record objections in the same audit trail as crawl events, and action them through the erasure workflow rather than by hand.

An objection under legitimate interests is not a request you weigh at leisure; absent compelling grounds, processing stops. Building the objection path as a first-class workflow — resolve the subject, suppress future collection, erase what is held, confirm — is what makes the mitigation you claimed in the assessment real.

Verification & Testing #

Each is cheaper to fix while writing than after a complaint.

def test_registry_entries_reference_a_current_assessment(registry):
    for entry in registry:
        if not entry.personal_data_expected:
            continue
        assessment = load_assessment(entry.assessment_ref)
        assert_current(assessment)
        assert_registry_matches(entry, assessment)

def test_extractor_fields_are_within_the_assessment(registry, extractors):
    for entry in registry:
        if not entry.personal_data_expected:
            continue
        controls = load_assessment(entry.assessment_ref)["controls"]
        fields = set(extractors[entry.host].field_names())
        assert not (fields & set(controls["excluded_fields"]))

The second test reads the actual extractor rather than the registry’s declaration of what it extracts, which closes the gap between what the configuration claims and what the code does.

Compliance & Operational Guardrails #

No host with personal_data_expected may be crawled without an approved, current assessment.
The assessment’s controls generate the registry entry; the two can never diverge silently.
An overdue review expires the basis and stops the crawl for that host.
Objections are recorded, actioned through the erasure workflow, and counted as a metric.
Each assessment version is retained after supersession, so past processing can be explained.

Common Mistakes #

Writing the assessment after the crawl started. The basis has to exist before the processing does; a retrospective document is evidence of the gap, not a cure for it.
Claiming mitigations that are not implemented. Pseudonymisation and retention limits stated in an assessment and absent from the pipeline are the fastest way to turn a defensible position into an indefensible one.
Deleting superseded versions. Explaining processing that happened last year requires the assessment that was in force last year.

Frequently Asked Questions #

Who has to approve the assessment? #

Whoever in your organisation owns data-protection decisions — a DPO where one exists, otherwise a named accountable person. The engineering value of naming them in the file is that the approver is unambiguous and the record survives staff changes. What matters is that it is not the same person who wants the data, reviewing their own request.

How often should it be reviewed? #

Six months is a sensible default, and immediately whenever the purpose, the volume, the categories of data, or the sources change materially. Tie the review date into the same expiry mechanism as the crawl registry so an overdue review has an operational consequence rather than being a diary entry.

Does the assessment need to be published? #

The assessment itself, no. The substance of it — what you collect, why, on what basis, for how long, and how to object — belongs in a public processing notice, because transparency is a separate obligation that pseudonymisation and minimisation do not satisfy. Keep the notice versioned alongside the assessment so the two cannot drift apart.

What makes an assessment fail its own balancing test? #

Usually one of four things. The purpose is stated so broadly that necessity cannot be evaluated — “improve our products” justifies any collection and therefore justifies none. The alternatives section lists no alternatives, which means necessity was assumed rather than tested. The impact assessment considers only the risk to your organisation rather than to the individual. Or the mitigations are aspirational: retention limits nobody enforces, pseudonymisation nobody implemented, an objection route that returns a 404. Reviewers spot all four quickly, and each is cheaper to fix while writing the assessment than after a complaint.

Does the assessment need to cover each host separately? #

Not necessarily. One assessment can cover a class of processing across many similar sources — public trade listings across twenty marketplaces, for instance — provided the categories of data, the purpose and the impact are genuinely the same. Where a source differs materially, such as one carrying contact details the others do not, it needs its own assessment or an explicit amendment. Registry entries reference an assessment by identifier, so the many-to-one mapping is expressed in configuration and stays visible.

GDPR Compliance for Scraped Personal Data — the obligations this assessment sits inside.
Pseudonymising Scraped Personal Data at Ingest — implementing the mitigation the balancing relies on.
Enforcing Retention Windows With Scheduled Jobs — making the assessed retention period real.