Crawl Observability with Prometheus and Grafana #

A crawler you cannot see is a crawler you cannot trust to behave. Crawl observability with Prometheus and Grafana turns a black-box scraping fleet into a set of named, queryable time series — request rate, error rate, 429 counts, latency distribution, deduplication ratio, and queue depth — so you can prove the crawl is both healthy and polite. This technique belongs to the Pipeline Storage, Deduplication & Monitoring section, and it solves a problem every team hits at scale: an impolite crawler often looks fine from the inside until the target starts blocking, and by then you have burned goodwill and proxy budget. Instrumentation surfaces the early signals — a climbing 429 rate, a widening latency tail — while they are still cheap to fix. Data engineers, SREs, and compliance officers all read the same dashboards: engineers to keep throughput up, compliance to demonstrate the crawl respected the target’s limits.

Core Principles & Metric Foundation #

Prometheus is a pull-based system. Your crawler exposes a /metrics HTTP endpoint holding the current value of each metric; a Prometheus server scrapes that endpoint on a fixed interval and stores the samples as time series. Grafana then queries Prometheus with PromQL to render dashboards, and Alertmanager fires when a rule’s condition holds. The mental model that matters: your process only ever reports current state (a counter’s running total, a gauge’s instantaneous value); all rates, percentiles, and trends are computed at query time from the scraped history.

Four metric types cover everything a crawl needs. A counter only ever increases — total requests, total errors, total 429s. A gauge goes up and down — queue depth, in-flight requests, active proxies. A histogram buckets observations to let you compute percentiles — request latency. A summary is a client-side percentile, used rarely. Choosing correctly is the whole game: counting 429s in a gauge would erase the history you need to alert on.

The Metrics That Reveal Crawl Health #

The 429 counter deserves special weight. A healthy, polite crawler produces near-zero 429s because it is already pacing itself with an adaptive token-bucket rate limiter. A rising crawl_rate_limited_total is the metric that proves you are pushing a target harder than it wants — the observability counterpart to the politeness controls themselves.

Labels and Cardinality #

Labels slice a metric by dimension — per host, per status, per queue. They are powerful and dangerous: every distinct label-value combination is a separate time series, so a label with unbounded values (a full URL, a request ID) explodes Prometheus memory. Label by target host, status class, and queue name; never by URL or by scraped identifier.

Implementation Steps #

1. Instrument the crawler with the Prometheus client #

Define the metrics once at module load, then update them at the exact points requests happen. Exposing these cleanly is covered in depth in exposing scraper metrics with the Prometheus client; the core instrumentation is:

from prometheus_client import Counter, Gauge, Histogram, start_http_server
import time, httpx

REQUESTS = Counter(
    "crawl_requests_total", "HTTP requests issued", ["host", "status"],
)
RATE_LIMITED = Counter(
    "crawl_rate_limited_total", "429 responses received", ["host"],
)
LATENCY = Histogram(
    "crawl_request_duration_seconds", "Request latency", ["host"],
    buckets=(0.1, 0.25, 0.5, 1, 2, 5, 10, 30),
)
QUEUE_DEPTH = Gauge("crawl_queue_depth", "URLs waiting to be fetched", ["queue"])
IN_FLIGHT = Gauge("crawl_in_flight", "Concurrent in-flight requests")
DEDUP_RATIO = Gauge("crawl_dedup_ratio", "Fraction of URLs already seen", ["dataset"])

def fetch(client: httpx.Client, url: str, host: str) -> httpx.Response:
    IN_FLIGHT.inc()
    start = time.perf_counter()
    try:
        resp = client.get(url)
        REQUESTS.labels(host=host, status=str(resp.status_code)).inc()
        if resp.status_code == 429:
            # This counter is the audit-grade signal that we hit a target's limit.
            RATE_LIMITED.labels(host=host).inc()
        return resp
    finally:
        LATENCY.labels(host=host).observe(time.perf_counter() - start)
        IN_FLIGHT.dec()

start_http_server(9091)   # exposes /metrics for Prometheus to scrape

2. Report the deduplication ratio and queue depth #

The dedup ratio and backlog come from the pipeline’s own state, refreshed on a timer rather than per request:

def refresh_pipeline_gauges(redis, queue_name: str, dataset: str):
    QUEUE_DEPTH.labels(queue=queue_name).set(redis.llen(f"frontier:{queue_name}"))
    seen = redis.scard(f"dedup:{dataset}")
    fetched = int(redis.get(f"fetched:{dataset}") or 1)
    # Wasted-work signal: high ratio means the frontier keeps re-surfacing known URLs.
    DEDUP_RATIO.labels(dataset=dataset).set(seen / max(fetched, 1))

3. Point Prometheus at the crawler #

The scrape config tells Prometheus where and how often to pull. Keep the interval short enough to catch a 429 spike but not so short it perturbs the crawler:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - crawl_alerts.yml
scrape_configs:
  - job_name: crawler-fleet
    static_configs:
      - targets: ["crawler-1:9091", "crawler-2:9091"]
    metric_relabel_configs:
      # Drop any accidental high-cardinality label before it is stored.
      - source_labels: [url]
        action: labeldrop

4. Build the Grafana dashboard #

Grafana panels are PromQL queries. The rate of a counter over a window is the fundamental building block — full panel construction is walked through in building a Grafana dashboard for crawl health. The queries that carry a dashboard:

# Request rate per host (requests/sec, 5-min window)
sum by (host) (rate(crawl_requests_total[5m]))

# Error rate as a fraction of all requests
sum(rate(crawl_requests_total{status=~"5.."}[5m]))
  / sum(rate(crawl_requests_total[5m]))

# 429s per minute per host — the politeness panel
sum by (host) (rate(crawl_rate_limited_total[5m])) * 60

# p95 request latency from the histogram
histogram_quantile(0.95, sum by (le, host) (rate(crawl_request_duration_seconds_bucket[5m])))

Error Handling & Observability #

Instrumentation itself must fail safe. A metrics endpoint that crashes should never take the crawler down, and a scrape that misses should be visible rather than silent.

Encode SLOs as alerting rules so the dashboard is not something a human must watch:

# crawl_alerts.yml
groups:
  - name: crawl-slos
    rules:
      - alert: HighRateLimitedRatio
        expr: sum(rate(crawl_rate_limited_total[5m]))
              / sum(rate(crawl_requests_total[5m])) > 0.02
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "Over 2% of requests are being 429'd — crawler is too aggressive"
      - alert: ErrorBudgetBurn
        expr: sum(rate(crawl_requests_total{status=~"5.."}[5m]))
              / sum(rate(crawl_requests_total[5m])) > 0.05
        for: 5m
        labels: {severity: critical}
      - alert: QueueBacklogGrowing
        expr: deriv(crawl_queue_depth[15m]) > 0 and crawl_queue_depth > 10000
        for: 20m
        labels: {severity: warning}

The HighRateLimitedRatio alert is the one compliance cares about most: it converts “are we crawling politely?” from a judgement call into a threshold. When it fires, the fix usually lives in the pacing layer — tightening the delays described in implementing polite rate limiting rather than in the observability stack.

Compliance Boundaries #

Observability is where a scraping operation demonstrates good faith. The 429 and latency panels are contemporaneous evidence that you monitored the target’s signals and responded — directly relevant to a CFAA “authorized access” posture and to honouring a site’s Terms of Service. Retaining dashboards and alert history gives a compliance officer an audit trail that the crawl stayed inside agreed limits. GDPR intersects here only lightly: metrics must not embed personal data in labels, which the host-and-status-only labelling already guarantees.

• crawl_rate_limited_total `crawl_rate_limited_total` is instrumented and has an SLO alert under 2–3%.
• Latency is a histogram with buckets that actually cover the observed range.
• No metric is labelled by URL, request ID, or any scraped personal identifier.
• Error-rate and queue-depth alerts route to an on-call channel, not just a dashboard.
• Alert and dashboard history is retained long enough to serve as a compliance record.
• A 429 A `429` spike triggers a review of the rate-limiter config, not just a proxy swap.

Budget maths: if a target publishes a limit of L requests/minute across your whole fleet, set the HighRateLimitedRatio alert well below the point where sustained request rate — sum(rate(crawl_requests_total[5m])) * 60 — approaches L. The dashboard should show you crossing 80% of L long before the target starts refusing.

Common Mistakes #

1. Counting events in a gauge. Using a gauge for 429s or errors destroys the history rate() needs, so you can never alert on a spike. Right: events are counters, states are gauges.
2. Labelling by URL or request ID. Unbounded label values create one time series per value and exhaust Prometheus memory. Right: label only by bounded dimensions — host, status class, queue.
3. Histogram buckets that miss the real latency range. If every request lands in the last bucket, histogram_quantile returns garbage. Right: choose buckets spanning your actual p50 to p99.
4. Alerting on raw counter values. A counter resets to zero on every deploy; a threshold on the raw value fires spuriously. Right: alert on rate() or increase(), which are reset-aware.
5. Watching dashboards instead of encoding SLOs. A 429 spike at 3am helps no one if it only lives on a screen. Right: every health signal that matters gets an Alertmanager rule with a severity and a route.

Frequently Asked Questions #

What scrape interval should I use for a crawler? #

Start at 15 seconds. That is frequent enough to catch a 429 or latency spike within a couple of samples while adding negligible load — a /metrics scrape is a single cheap HTTP GET. Going below 5 seconds rarely helps and multiplies Prometheus storage. If you need finer resolution for a specific investigation, raise it temporarily rather than running the whole fleet hot. The for: duration on your alerts matters more than the scrape interval for avoiding false pages.

Which single metric best tells me my crawler is being impolite? #

The ratio of crawl_rate_limited_total to crawl_requests_total. A polite crawler that paces itself correctly produces almost no 429s, so any sustained rate-limited fraction above a couple of percent means you are requesting faster than the target tolerates. Watch it per host — one aggressive target host can hide inside a healthy fleet-wide average, which is why the panel and the SLO alert both group by host.

Counter, gauge, or histogram — how do I choose? #

Ask what the value does over time. If it only ever grows and you care about its rate (requests, errors, 429s), use a counter. If it moves up and down and you care about its current level (queue depth, in-flight requests, dedup ratio), use a gauge. If you need percentiles of a distribution (latency), use a histogram and let PromQL compute quantiles from the buckets at query time. Picking the wrong type usually means you cannot ask the question you actually have later.

Do I need Grafana, or is Prometheus enough? #

Prometheus can store, query, and alert entirely on its own — Alertmanager handles paging without Grafana in the loop. What Grafana adds is human-readable dashboards for exploration and for sharing crawl health with non-engineers, including compliance reviewers. Many teams run alerting purely in Prometheus/Alertmanager and use Grafana as the read-only window onto the same data. Neither one stores anything the other cannot; they are complementary layers over the same time series.

Related guides #

• Pipeline Storage, Deduplication & Monitoring — the section this observability technique belongs to.
• Building a Grafana dashboard for crawl health — panel-by-panel construction of the dashboard sketched here.
• Exposing scraper metrics with the Prometheus client — the instrumentation and /metrics endpoint in detail.
• Implementing polite rate limiting — the controls that keep the 429 panel flat.
• Exponential backoff and retry logic — how the crawler reacts to the errors these metrics count.