Compliant Web Scraping

Alerting on Crawl Success Rate #

Success rate is the metric most crawl teams alert on first and configure worst. A global threshold pages somebody every time one large host has a bad afternoon, and it stays silent when a small but important host stops answering entirely. This guide builds alerts that fire on the conditions a person can act on, as part of crawl observability with Prometheus and Grafana in the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

Three properties make crawl success rate awkward to alert on.

One global threshold versus per-host baselinesComparing a host with itself is what makes one threshold work everywhere.One global threshold versus per-host baselinesGlobal success thresholdLarge hosts dominate the numberA small host can go dark unseenTuned until it never firesNobody can act on itPer-host, versus yesterdayEach host judged on its own historyA 99 to 90 drop is caughtLow-traffic hosts are guardedEvery alert names a host
Comparing a host with itself is what makes one threshold work everywhere.

It is not one number. A crawl touching eight hundred hosts has eight hundred success rates, and the aggregate is dominated by whichever host has the most traffic. A host contributing 1% of requests can go entirely dark without moving the global figure past its threshold.

Failure is normal. 404s are a routine outcome of crawling — links rot, listings expire — and counting them as failures makes the baseline both high and meaningless.

The interesting signal is often a change rather than a level. A host that has always succeeded 70% of the time is fine; one that dropped from 99% to 90% this morning is not, even though the second number is higher.

The alerts that work therefore operate per host, exclude expected outcomes, and compare against recent behaviour rather than an absolute constant.

Step-by-Step Implementation #

1. Instrument outcomes, not raw statuses #

Four conditions worth alerting onThe last condition is invisible to every ratio-based alert.Four conditions worth alerting onConditionSeverityFirst actionSuccess rate below its baselinePageCheck outcome mixAny refusal observedPagePause and reviewThrottling persists an hourTicketLower the rateA host stopped producing trafficTicketCheck the frontier
The last condition is invisible to every ratio-based alert.
from prometheus_client import Counter

CRAWL_OUTCOMES = Counter(
    "crawl_outcomes_total", "Crawl request outcomes",
    ["host", "outcome"],        # ok | not_found | throttled | transient | refused | parse_error
)

EXPECTED = frozenset({"ok", "not_found"})       # not_found is normal, not a failure

def record_outcome(host: str, outcome: str) -> None:
    CRAWL_OUTCOMES.labels(host=host_label(host), outcome=outcome).inc()

Using the classifier’s outcome rather than the HTTP status is what makes the alert meaningful. A 404 and a 403 are both non-200 and mean entirely different things; collapsing them into “error” produces an alert nobody can act on without first opening the logs.

Keep the host label bounded — registry hosts by name, everything else collapsed into other — or the series count grows with the crawl frontier.

2. Rules that fire on the right conditions #

groups:
  - name: crawl-health
    interval: 1m
    rules:
      # Reusable: success share per host over 30 minutes.
      - record: crawl:success_ratio:30m
        expr: |
          sum by (host) (rate(crawl_outcomes_total{outcome=~"ok|not_found"}[30m]))
            /
          sum by (host) (rate(crawl_outcomes_total[30m]))

      # 1. A tracked host has collapsed relative to its own recent baseline.
      - alert: CrawlSuccessRateDropped
        expr: |
          crawl:success_ratio:30m
            < 0.7 * (crawl:success_ratio:30m offset 1d)
          and sum by (host) (rate(crawl_outcomes_total[30m])) > 0.05
        for: 15m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.host }} success rate fell to {{ $value | humanizePercentage }}"
          runbook: "https://example.org/runbooks/crawl-success-drop"

      # 2. Any refusal at all is a compliance signal, not a reliability one.
      - alert: CrawlRefusalObserved
        expr: increase(crawl_outcomes_total{outcome="refused"}[1h]) > 0
        for: 0m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.host }} refused the crawler — pause and review"

      # 3. Throttling that persists means the pace is still wrong.
      - alert: CrawlPersistentlyThrottled
        expr: |
          sum by (host) (rate(crawl_outcomes_total{outcome="throttled"}[1h]))
            / sum by (host) (rate(crawl_outcomes_total[1h])) > 0.02
        for: 30m
        labels: {severity: ticket}

      # 4. A host that has simply stopped producing requests.
      - alert: CrawlHostSilent
        expr: |
          sum by (host) (rate(crawl_outcomes_total[1h])) == 0
          and sum by (host) (rate(crawl_outcomes_total[1h] offset 1d)) > 0.05
        for: 1h
        labels: {severity: ticket}

The first rule compares each host against its own value yesterday, which is what makes one threshold work across hosts with very different baselines. The traffic guard prevents a host with three requests an hour from paging anyone on the strength of a single failure.

The second rule has no threshold and no for duration deliberately. A refusal that names the crawler is a message from a site operator, and the right response time is minutes rather than “once it becomes a trend”.

The fourth catches the failure that success-rate alerting misses entirely: a host producing no requests has a success rate of nothing at all, and every ratio-based alert stays silent while the crawl quietly stops covering it.

3. Suppress the noise you can predict #

inhibit_rules:
  # A refusal explains the success drop; page once, not twice.
  - source_matchers: [alertname="CrawlRefusalObserved"]
    target_matchers: [alertname="CrawlSuccessRateDropped"]
    equal: [host]

  # Our own outage is not a host problem.
  - source_matchers: [alertname="EgressUnavailable"]
    target_matchers: [alertname=~"Crawl.*"]

The second inhibition matters more than it looks. When your own egress fails, every host’s success rate collapses simultaneously, and without inhibition the on-call engineer receives eight hundred pages describing one problem.

A simultaneous drop across unrelated hosts is itself diagnostic — it is nearly always your side — so it is worth a separate alert on the count of hosts currently in a drop state, rather than on the hosts individually.

4. Write the runbook alongside the rule #

Every alert needs a documented first action, and if that cannot be written the alert should be a dashboard panel instead.

## CrawlSuccessRateDropped

**First checks, in order:**
1. Is `CrawlRefusalObserved` also firing for this host? → follow the refusal runbook; stop here.
2. Is our egress healthy? Check `EgressUnavailable` and the pool health dashboard.
3. What is the outcome breakdown? `sum by (outcome) (rate(crawl_outcomes_total{host="…"}[30m]))`
   - mostly `transient` → host-side instability; the circuit breaker should already be reducing load
   - mostly `throttled` → our pace is too high; reduce the host rate and confirm it took effect
   - mostly `parse_error` → the site changed; check selector coverage and the fixture corpus
4. If unclear, pause the host and open a ticket. A paused host costs throughput; a misread refusal costs the relationship.

Step four is the important one. Pausing is always a safe action, and making that explicit removes the hesitation that leads to a crawl continuing through a signal nobody had time to interpret.

Verification & Testing #

Alert hygieneWithout the fourth rule, one egress fault pages once per host.Alert hygiene✓404 counts as success, not as failure✓Low-traffic hosts cannot page on a single failure✓A refusal inhibits the redundant success-rate page✓Our own egress outage inhibits every host alert✓Every alert names an owner and a first action
Without the fourth rule, one egress fault pages once per host.
def test_not_found_counts_as_success():
    for _ in range(50):
        record_outcome("example.com", "not_found")
    assert success_ratio("example.com") == 1.0

def test_low_traffic_host_does_not_page():
    record_outcome("tiny.example", "transient")
    assert not alert_would_fire("CrawlSuccessRateDropped", host="tiny.example")

def test_refusal_pages_immediately():
    record_outcome("example.com", "refused")
    assert alert_would_fire("CrawlRefusalObserved", host="example.com", after_seconds=0)

def test_silent_host_is_caught():
    seed_history("example.com", requests_per_second=1.0, days_ago=1)
    advance_time(hours=2)                       # no requests since
    assert alert_would_fire("CrawlHostSilent", host="example.com")

Test the rules, not just the metrics. Prometheus provides a unit-test harness for alerting rules, and encoding the four cases above prevents a rule change from silently disabling an alert — a failure mode that is invisible precisely because nothing fires.

Review the alert history monthly. An alert that has never fired is either testing nothing or is inhibited by another; one that fires weekly and is always acknowledged without action is training people to ignore the channel.

Compliance & Operational Guardrails #

  • Refusals page immediately and have no threshold, because they are compliance signals.
  • Alert labels carry no URLs, identifiers or personal data.
  • Every alert names an owner and a runbook with a first action.
  • Pausing a host is always an acceptable response and is documented as such.
  • Retention-job failures alert on the same channel as crawl health, not a separate one nobody watches.

Common Mistakes #

  1. A single global success-rate alert. Large hosts mask small ones and the number means nothing.
  2. Counting 404 as failure. The baseline becomes high and noisy, so the alert is tuned until it never fires.
  3. No alert for a host that stopped producing traffic. Ratio-based alerts are silent on zero denominators.

Frequently Asked Questions #

How long should the for duration be? #

Ten to fifteen minutes for reliability alerts, because crawl metrics are noisier than application metrics — response times depend on hosts you do not control. Zero for compliance signals such as refusals, where the value of the alert is its immediacy.

Should alerts be per host or per host group? #

Per host for the registry hosts that matter, aggregated for the remainder. Grouping by section or by data source gives a middle tier that catches a systemic problem affecting several small hosts without producing a page per host.

What about alerting on data quality rather than crawl health? #

Worth doing and worth separating. Quarantine rate, selector coverage and field null rates belong on their own alerts with their own owner, because a crawl that fetches perfectly and extracts nothing looks completely healthy on every metric in this guide. Both sets are covered in the observability topic.

How should alerts behave during a planned crawl pause? #

Silence them explicitly rather than letting them fire and be acknowledged. A maintenance silence scoped to the affected hosts, with an expiry, keeps the channel meaningful — and an expiring silence forces someone to confirm the pause is over. Blanket silences with no expiry are how a crawl ends up unmonitored for a month after a two-hour maintenance window.

Should data-quality alerts page or ticket? #

Ticket, almost always. A selector that stopped matching produces no more damage at three in the morning than at nine, because the pages are still being fetched and the raw responses retained. Compliance signals — a refusal, a retention job failure — page, because both have consequences that grow with the delay. Getting that split right is what keeps people responding to the pages that matter.