Compliant Web Scraping

Building a Grafana dashboard for crawl health #

A crawl fleet that emits metrics but has nowhere to read them is effectively blind, so this page walks through assembling a single Grafana dashboard that answers one question at a glance: is the crawler healthy right now? It is a focused how-to within Crawl Observability with Prometheus and Grafana, part of the broader Pipeline Storage, Deduplication & Monitoring section. Where the parent technique page covers the full instrumentation-to-alerting loop, here we concentrate narrowly on panel layout, the exact PromQL behind each panel, and provisioning the finished board as version-controlled JSON so it deploys identically across staging and production.

Problem Framing #

Once a distributed crawler runs across dozens of workers, raw log tailing stops scaling. Operators need aggregate rates, not individual events: a spike in 429 responses signals you are outrunning a target’s rate limits; a climbing queue depth means fetch workers are falling behind the frontier; a collapsing dedup ratio hints that a URL canonicalizer regressed and you are re-fetching the same pages. Each of these is invisible in logs but obvious on a time-series panel.

The trap is building a dashboard nobody reads. A crawl-health board earns a place on the wall only if every panel maps to an action. We therefore pick six signals that each trigger a distinct operator response, drive them all from metrics exposed by the Python prometheus_client, and commit the dashboard definition to the same repository as the crawler.

Step-by-Step Implementation #

  1. Confirm the metric names. The queries below assume a scrape_requests_total{status,host} counter, a scrape_request_duration_seconds histogram, a scrape_queue_depth gauge, and a scrape_items_deduped_total / scrape_items_total counter pair. Keep label cardinality low — host and status only, never full URLs.
  2. Build the six core panels. Add each panel as a time series (or stat) visualization and paste the PromQL from step 3. Use a rate() window of 5m to smooth per-scrape noise while staying responsive.
  3. Write the PromQL. These are the exact expressions each panel runs:
# Request rate (req/s), broken out by target host
sum by (host) (rate(scrape_requests_total[5m]))

# Error rate as a fraction of all requests (5xx + connection errors)
sum(rate(scrape_requests_total{status=~"5..|error"}[5m]))
  / sum(rate(scrape_requests_total[5m]))

# 429 rate specifically — your politeness early-warning signal
sum by (host) (rate(scrape_requests_total{status="429"}[5m]))

# p95 fetch latency from the histogram buckets
histogram_quantile(0.95,
  sum by (le, host) (rate(scrape_request_duration_seconds_bucket[5m])))

# Current frontier queue depth (gauge, no rate needed)
sum(scrape_queue_depth)

# Deduplication ratio: share of fetched items dropped as duplicates
sum(rate(scrape_items_deduped_total[5m]))
  / sum(rate(scrape_items_total[5m]))
  1. Add threshold coloring. On the 429-rate and error-rate panels, set field thresholds so the panel turns amber then red as it climbs — a red 429 panel is the cue to throttle before a target bans you.
  2. Provision the dashboard as code. Rather than clicking panels into existence on every environment, drop a provider YAML file into Grafana’s provisioning/dashboards/ directory that points at a folder of exported JSON:
# /etc/grafana/provisioning/dashboards/crawl-health.yaml
apiVersion: 1
providers:
  - name: crawl-health
    orgId: 1
    folder: Crawling
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false        # dashboard is code; edits happen in git
    options:
      path: /var/lib/grafana/dashboards/crawl
      foldersFromFilesStructure: true
  1. Export and commit the JSON. Build the board once in the UI, then use Share → Export → Save to file (or the /api/dashboards/uid/<uid> endpoint) to produce a JSON model. A trimmed panel from that model looks like this:
{
  "title": "Crawl Health",
  "uid": "crawl-health",
  "panels": [
    {
      "title": "429 rate by host",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum by (host) (rate(scrape_requests_total{status=\"429\"}[5m]))",
          "legendFormat": ""
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "reqps",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "orange", "value": 0.2 },
              { "color": "red", "value": 1 }
            ]
          }
        }
      }
    }
  ],
  "schemaVersion": 39,
  "refresh": "30s"
}

Commit this file to your crawler repo so the dashboard ships in the same pull request as the metric that feeds it.

Verification & Testing #

Before trusting the board, verify each query returns data directly from Prometheus. Hit the HTTP API and confirm a non-empty result array:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=sum by (host) (rate(scrape_requests_total[5m]))' \
  | jq '.data.result | length'

A return of 0 means the metric name or label is wrong, or no scrapes have landed in the window. To exercise the provisioning path, restart Grafana with the YAML in place and check the log for finished to provision dashboards; the board should appear in the Crawling folder without any manual import. Finally, drive synthetic load through the crawler against a local httpbin returning 429s and watch the 429 panel cross its amber threshold in real time.

Compliance & Operational Guardrails #

  • Keep the 429-rate panel on the primary row and wire an alert to it: a sustained 429 rate is direct evidence you are exceeding a host’s tolerance, and honoring it is core to running an adaptive rate limiter from response headers.
  • Never label metrics with full request URLs or query strings — they can carry personal data and will explode Prometheus cardinality. Aggregate to host so the dashboard reveals no individual-level browsing detail.
  • Treat the dashboard JSON as auditable infrastructure: because it is version-controlled, you can prove to a compliance reviewer exactly which health signals were monitored on any past date.

Common Mistakes #

  1. Graphing _total counters raw. Counters only ever climb; without rate() you get a meaningless upward ramp instead of a per-second signal.
  2. Computing p95 with avg over the _sum/_count pair. That yields a mean, not a percentile — use histogram_quantile over _bucket series so tail latency stays visible.
  3. Leaving allowUiUpdates: true. Operators then hand-edit panels that a redeploy silently reverts, and the git copy drifts out of sync with reality.

Frequently Asked Questions #

How do I show one panel per target host without editing PromQL for each? #

Use a Grafana dashboard variable (for example $host) populated by label_values(scrape_requests_total, host), then reference $host inside the query and enable Repeat panel on that variable. Grafana renders one panel per host automatically as new targets appear.

Should queue depth be a gauge or derived from a counter? #

Queue depth is instantaneous state, so expose it as a gauge and graph it directly — no rate(). If you also want throughput, add separate enqueue and dequeue counters and rate those; the gauge answers “how deep now?” while the counters answer “how fast is it draining?”.

Why does my dedup-ratio panel briefly read above 1.0? #

That is a counter-reset artifact when a worker restarts and its _deduped_total and _total series re-initialize at slightly different scrape moments. Clamp it with clamp_max(<expr>, 1) for display, and it settles once both counters advance together.