Compliant Web Scraping

Exposing scraper metrics with the Python prometheus_client #

Before any Grafana panel can plot crawl health, your scraper has to publish numbers Prometheus can scrape, and this page covers exactly that instrumentation layer using the official Python prometheus_client. It sits under Crawl Observability with Prometheus and Grafana within the Pipeline Storage, Deduplication & Monitoring section, and stays deliberately narrow: which metric type to reach for, how to label without wrecking cardinality, and how to stand up a /metrics endpoint that a Prometheus server can poll. The companion guide on building a Grafana dashboard for crawl health consumes precisely the series we define here.

Problem Framing #

Scrapers fail in ways that are quiet until they are catastrophic: latency creeps up as a target adds bot defenses, error rates climb after a markup change, and retry loops burn budget without anyone noticing. Instrumenting the fetch loop with counters, histograms, and gauges turns those silent failures into queryable time series long before they page you at 3 a.m.

The decision that trips up most engineers is not whether to add metrics but how to shape their labels. Prometheus creates one independent time series per unique label-value combination, so a single ill-chosen label — say, the full request URL — can generate millions of series and knock over your monitoring stack faster than the scraper itself ever could. The rest of this page is about instrumenting the right signals with disciplined, low-cardinality labels.

Step-by-Step Implementation #

  1. Pick the right metric type per signal. Use a Counter for monotonically increasing totals (requests sent, items parsed, retries), a Histogram for distributions you will percentile later (fetch latency, response size), and a Gauge for values that go up and down (in-flight requests, frontier queue depth).
  2. Define metrics once, at module scope. Metric objects register into a global collector on creation, so instantiate them at import time — never inside the request loop, or you will hit Duplicated timeseries errors.
  3. Choose bounded labels. Label by host and coarse status class only. Never label by URL, user ID, or timestamp.
  4. Instrument the fetch path with .labels(...).inc(), .observe(), and gauge context managers.
  5. Expose /metrics via start_http_server on a dedicated port so Prometheus can scrape it.
import time
import random
from urllib.parse import urlsplit
from prometheus_client import Counter, Histogram, Gauge, start_http_server
import httpx

# --- Metric definitions (module scope, registered once) ---
REQUESTS = Counter(
    "scrape_requests_total",
    "Total HTTP requests issued by the crawler",
    ["host", "status"],          # low-cardinality labels only
)
LATENCY = Histogram(
    "scrape_request_duration_seconds",
    "Fetch latency in seconds",
    ["host"],
    buckets=(0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)
IN_FLIGHT = Gauge(
    "scrape_in_flight_requests",
    "Requests currently awaiting a response",
)

def status_class(code: int) -> str:
    # Collapse to 2xx/4xx/5xx-style buckets, but keep 429 explicit
    return "429" if code == 429 else f"{code // 100}xx"

def fetch(client: httpx.Client, url: str) -> httpx.Response | None:
    host = urlsplit(url).hostname or "unknown"
    IN_FLIGHT.inc()
    start = time.perf_counter()
    try:
        resp = client.get(url, timeout=10)
        LATENCY.labels(host=host).observe(time.perf_counter() - start)
        REQUESTS.labels(host=host, status=status_class(resp.status_code)).inc()
        return resp
    except httpx.RequestError:
        REQUESTS.labels(host=host, status="error").inc()
        return None
    finally:
        IN_FLIGHT.dec()

if __name__ == "__main__":
    # Serve /metrics on :8000 for Prometheus to scrape.
    # Bind to loopback or an internal interface — never expose publicly.
    start_http_server(8000, addr="127.0.0.1")
    with httpx.Client(headers={"User-Agent": "compliant-crawler/1.0 (+https://example.com/bot)"}) as client:
        for url in ["https://httpbin.org/status/200", "https://httpbin.org/status/429"]:
            fetch(client, url)
            time.sleep(random.uniform(1, 2))  # polite pacing between requests
        time.sleep(3600)  # keep the metrics endpoint alive for scraping

Point a Prometheus scrape job at 127.0.0.1:8000 and the series above start flowing. The Gauge also works as a decorator or context manager — with IN_FLIGHT.track_inprogress(): — if you prefer that idiom.

Verification & Testing #

The fastest check is to curl the endpoint and confirm your metric families render in the text exposition format:

curl -s http://127.0.0.1:8000/metrics | grep '^scrape_'

You should see scrape_requests_total, the histogram’s _bucket/_sum/_count families, and scrape_in_flight_requests. For a unit test, prometheus_client exposes helpers so you can assert a counter moved without standing up an HTTP server:

from prometheus_client import REGISTRY

def test_request_counter_increments():
    before = REGISTRY.get_sample_value(
        "scrape_requests_total", {"host": "httpbin.org", "status": "2xx"}
    ) or 0
    fetch(client, "https://httpbin.org/status/200")
    after = REGISTRY.get_sample_value(
        "scrape_requests_total", {"host": "httpbin.org", "status": "2xx"}
    )
    assert after == before + 1

To confirm the histogram is usable for percentiles, query histogram_quantile(0.95, rate(scrape_request_duration_seconds_bucket[5m])) in Prometheus and check it returns a bounded number rather than NaN.

Compliance & Operational Guardrails #

  • Bind the metrics server to loopback or an internal network interface. /metrics can leak target hostnames and traffic volumes, so treat it as internal telemetry, not a public endpoint.
  • Keep every label free of personal data. Because a label value spawns a permanent time series, putting a scraped email, username, or full URL into one both breaches data-minimization expectations and creates an unbounded-cardinality outage risk.
  • Emit a 429-class counter so retry behavior stays observable, and feed it into your exponential backoff and retry logic so throttling decisions are driven by measured data rather than guesswork.

Common Mistakes #

  1. Labeling by URL or record identifier. This is the classic cardinality bomb — millions of one-off series that bloat memory and slow every query. Label by host and status class instead.
  2. Creating metric objects inside the loop. Re-registering the same name raises Duplicated timeseries in CollectorRegistry; define metrics once at module scope.
  3. Using a Gauge for something that only increases. Totals belong in a Counter so rate() and reset detection work; a gauge silently loses the reset semantics Prometheus relies on.

Frequently Asked Questions #

How do I expose metrics from a multiprocess or Gunicorn-based scraper? #

Set the PROMETHEUS_MULTIPROC_DIR environment variable to a writable directory and use prometheus_client.multiprocess.MultiProcessCollector. Each worker writes to shared mmap files that a single collector aggregates, so counters are not undercounted across forks. Without this, every worker reports only its own slice.

What histogram buckets should I choose for fetch latency? #

Buckets must bracket the latencies you actually care about. For web fetches, a spread like 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds captures fast responses through slow-timeout territory. Pick edges near your alerting thresholds, since histogram_quantile interpolates within a bucket and coarse edges blur the percentile.

Counter, Histogram, or Summary for latency? #

Prefer Histogram. A Summary computes quantiles inside each process and those quantiles cannot be aggregated across workers, whereas a Histogram ships raw buckets that Prometheus percentiles server-side — the only correct approach for a distributed crawl fleet.