Structured Logging and Log Shipping for Crawlers #

A crawler’s logs are two things at once: the fastest way to debug a broken parse, and the durable record that proves how a scrape behaved when someone asks months later. Structured logging and log shipping is the practice of emitting every log line as machine-parseable JSON, threading a correlation ID through the whole request lifecycle, redacting personal data before it is written, and streaming the result to a central store where it can be searched and retained. This technique is part of the Pipeline Storage, Deduplication & Monitoring section, and it solves a failure mode that free-text logging makes inevitable: you can grep a million lines of print() output for an error, but you cannot reliably answer “which crawl run touched this URL, under what authorization, and did it log any personal data it shouldn’t have?” Structured logs answer exactly that. The audience is broad — engineers debugging a stuck worker, SREs correlating a latency spike across services, and compliance officers who need a defensible audit trail.

Core Principles & Structured-Logging Foundation #

The shift from free text to structured logging is a shift from writing prose for humans to emitting events for machines that humans then query. A free-text line like WARN: retry 3 for /products failed with 503 forces a regex to extract anything. The same event as JSON — {"level":"warn","event":"retry_exhausted","url_path":"/products","status":503,"attempt":3} — is queryable by any field the moment it lands in a log store. Every field is a first-class filter, aggregation, and alert dimension.

Three properties make crawler logs useful rather than merely voluminous. Structure: one JSON object per line, stable field names. Correlation: a trace ID that ties every log from a single crawl request together, and a run ID that ties every request in a batch together. Discretion: no personal data or secrets in the payload, enforced before serialization, not hoped for afterward. Get these three right and the logs double as the compliance record described in the practices for documenting scraping authorization and data lineage.

A Canonical Log Schema #

Agree on the field set once and enforce it everywhere. A workable crawler schema:

The authorization field is what turns an operational log into an audit trail: every fetch records the basis on which it was permitted.

Correlation and Trace IDs #

A trace ID is generated once when a URL enters the pipeline and attached to every subsequent log line for that URL — fetch, parse, validate, store. Without it, logs from concurrent workers interleave into noise. With it, one query reconstructs a single request’s entire journey across services. A run ID sits one level up, grouping every request in a scheduled crawl so you can scope a search to “last night’s run of the marketplace crawl.”

Implementation Steps #

1. Emit JSON with a structured logging library #

Use a library that renders structured records natively rather than formatting strings by hand. structlog binds context once and carries it through:

import structlog, logging, sys, uuid

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,   # pulls in bound trace_id/run_id
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        redact_pii,                                 # custom processor, see step 3
        structlog.processors.JSONRenderer(),        # one JSON object per line
    ],
    logger_factory=structlog.PrintLoggerFactory(file=sys.stdout),
)
log = structlog.get_logger()

def crawl(url: str, host: str, run_id: str):
    trace_id = uuid.uuid4().hex
    structlog.contextvars.bind_contextvars(trace_id=trace_id, run_id=run_id, host=host)
    log.info("page_fetch_started", url_path=strip_query(url))
    # ... fetch and parse; every subsequent log line inherits trace_id/run_id ...

2. Bind the trace ID at the pipeline boundary #

Bind context at the single point where work enters, so no downstream call site has to remember to pass it. Emitting to stdout as JSON is deliberate: the process should not own log transport — the platform ships whatever the process prints.

def worker_loop(frontier, run_id: str):
    for url, host in frontier:
        try:
            crawl(url, host, run_id)
        except Exception:
            # exc_info renders a structured stack trace, still one JSON line
            log.error("crawl_failed", url_path=strip_query(url), exc_info=True)
        finally:
            structlog.contextvars.clear_contextvars()   # no context leaks between URLs

3. Redact PII before serialization #

Redaction must run as a processor in the logging pipeline, before the JSON is rendered, so there is no window in which raw personal data reaches disk. Redact by key (known sensitive field names) and by pattern (emails, tokens in query strings):

import re

SENSITIVE_KEYS = {"email", "phone", "name", "authorization_header", "cookie"}
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")

def redact_pii(logger, method_name, event_dict):
    for key in list(event_dict):
        if key in SENSITIVE_KEYS:
            event_dict[key] = "[REDACTED]"
        elif isinstance(event_dict[key], str):
            event_dict[key] = EMAIL_RE.sub("[REDACTED_EMAIL]", event_dict[key])
    return event_dict   # Compliance: PII never reaches the JSON renderer downstream.

def strip_query(url: str) -> str:
    return url.split("?", 1)[0]   # query strings routinely carry tokens/identifiers

4. Ship logs off the box with a forwarder #

The process writes JSON to stdout or a file; a dedicated forwarder tails it and streams to the central store. Filebeat and Vector are the common choices; Vector’s config for shipping to Elasticsearch and Loki side by side:

# vector.yaml — collect JSON logs, ship to Elasticsearch (search) and Loki (cheap retention)
sources:
  crawler_logs:
    type: file
    include: ["/var/log/crawler/*.json"]
sinks:
  elasticsearch:
    type: elasticsearch
    inputs: ["crawler_logs"]
    endpoints: ["https://es:9200"]
    bulk: {index: "crawler-%Y.%m.%d"}   # daily index eases retention/rollover
  loki:
    type: loki
    inputs: ["crawler_logs"]
    endpoint: "https://loki:3100"
    labels: {app: "crawler", run_id: ""}

The full Elasticsearch path — index templates, field mappings, and rollover — is detailed in shipping structured crawl logs to Elasticsearch.

Error Handling & Observability #

The logging path itself must never take down or block the crawler, and it must fail loudly enough that you notice logs stopped arriving.

The logging system deserves its own metrics, which live naturally alongside the crawler’s other signals in Prometheus and Grafana:

• log_lines_emitted_total{level} — a sudden drop means logging or shipping stalled.
• log_ship_lag_seconds — gap between emit and arrival in the store; alert above a minute.
• log_forwarder_buffer_bytes — rising toward the disk cap means the sink is down.
• redaction_failures_total — must stay at zero; any value is a potential PII leak.

Correlating a log_ship_lag_seconds spike with a crawler latency spike is exactly the kind of cross-signal analysis that structured logs plus crawl observability with Prometheus and Grafana make routine — the metric tells you when, the logs tell you why.

Compliance Boundaries #

Structured logs are the connective tissue of a scraping compliance story. The authorization field on every fetch, retained and searchable, is contemporaneous evidence of the basis on which each request was made — directly relevant to a CFAA authorized-access posture and to Terms-of-Service adherence. Under GDPR, logs are themselves a data store: if they contain personal data, they inherit the same storage-limitation and erasure obligations as any other sink, which is why redaction at source and a bounded retention window are non-negotiable.

• Every log line is valid JSON with the agreed schema fields present.
• A redaction processor runs before serialization and fails closed on error.
• URL query strings are stripped or redacted before logging.
• Every fetch records an authorization Every fetch records an `authorization` basis for the crawl.
• Log retention is bounded and documented, matching the crawl's data-retention policy.
• Shipping buffers to disk so a store outage never blocks the crawler.

Retention maths: pick a window that satisfies the longest legitimate need — typically 90 days for operational logs, longer only where an audit obligation requires it — and enforce it with index rollover so old logs age out automatically rather than accumulating as latent GDPR liability. If logs carry any residual personal data, that window is a hard ceiling, not a default.

Common Mistakes #

1. Logging raw request URLs with query strings. Tokens, session IDs, and email addresses ride in query parameters and land permanently in your log store. Right: strip the query and redact patterns before serialization.
2. Redacting after the fact. Scrubbing PII with a downstream cleanup job leaves a window where raw data sat on disk and in transit. Right: redact as a processor in the emit path, before the line is written.
3. No correlation ID. Without a trace ID, concurrent workers interleave into logs you cannot untangle. Right: bind a trace ID at the pipeline boundary and carry it through every line.
4. Letting the process own log transport. Shipping logs synchronously inside the crawler couples fetch throughput to log-store availability. Right: print JSON, let a forwarder buffer and ship independently.
5. Unbounded log retention. Keeping logs forever turns them into an ever-growing store of personal data with no lawful basis for the age. Right: set and enforce a documented retention window with automatic rollover.

Frequently Asked Questions #

Why JSON logs instead of formatted text with a good log format? #

Because every field becomes independently queryable the moment it lands. A well-formatted text line still forces a regex to extract the status code or the trace ID at search time, and that regex breaks the first time the format shifts. JSON gives the log store typed fields it can index, filter, and aggregate directly — you can compute an error rate per host, or pull every line for one trace_id, without parsing prose. The cost is slightly larger lines, which compression on the shipping path erases.

Where should PII redaction happen — in the app or in the log pipeline? #

As early as possible, which means inside the application’s own logging pipeline, as a processor that runs before the line is serialized and written. Redacting downstream — in the forwarder or the log store — means raw personal data already touched local disk and travelled over the network, which is exactly the exposure you are trying to prevent. Fail the redaction step closed: if the processor errors, drop the line rather than risk emitting un-redacted data.

Should I ship to Elasticsearch or Loki? #

They optimise for different things and pair well. Elasticsearch indexes every field, giving fast arbitrary search and aggregation, at higher storage and compute cost — ideal for the recent window you actively debug and audit against. Loki indexes only labels and stores the rest compressed, which is far cheaper for long-horizon retention where you rarely search but must keep the record. A common setup ships to both: Elasticsearch for the last few weeks, Loki for the months you retain for compliance.

Do crawler logs fall under GDPR? #

If they contain personal data, yes — logs are a data store like any other. A log line holding a scraped email address, a person’s name in a URL path, or any identifier is subject to the same storage-limitation and right-to-erasure obligations as your warehouse. The two defences are redaction at source, so logs hold as little personal data as possible, and a bounded, documented retention window enforced by automatic rollover so nothing lingers past its lawful basis.

Related guides #

• Pipeline Storage, Deduplication & Monitoring — the section this logging discipline belongs to.
• Shipping structured crawl logs to Elasticsearch — the index templates and rollover behind the shipping step here.
• Crawl observability with Prometheus and Grafana — the metrics layer that pairs with logs for cross-signal debugging.
• Documenting scraping authorization and data lineage — the audit-trail practice the authorization field feeds.
• Data retention and GDPR deletion workflows — the retention policy your log window must respect.