Circuit Breakers for Crawl Workers #

Backoff decides how long to wait before the next attempt; a circuit breaker decides whether to attempt at all. On a crawl spanning thousands of hosts, that second decision is what stops one unhealthy target from consuming an entire worker pool. This guide implements a per-host breaker with shared state across a fleet, as part of exponential backoff and retry logic in the Network Resilience & Proxy Management section.

Problem Framing #

Consider a crawl with fifty thousand queued URLs spread over eight hundred hosts, and one host that has gone down. Its URLs are interleaved through the queue, so workers pick them up continuously. Each one attempts, waits out the connect timeout, retries with backoff, and eventually fails — occupying a worker for perhaps ninety seconds to produce nothing. With five hundred URLs for that host, the crawl spends over twelve worker-hours on a target that is not answering.

Backoff bounds one URL’s attempts; only a breaker bounds the host.

Backoff does not help, because backoff operates within one URL’s attempt sequence and each URL starts fresh. The retry budget helps, but it caps waste rather than eliminating it. What is needed is a memory that spans URLs: this host is not answering, so do not try.

That is a circuit breaker, and the crawl-specific requirements are that it is keyed per host, shared across workers, and biased toward recovery — a host that comes back should be discovered within minutes, not at the next deploy.

Step-by-Step Implementation #

1. The state machine #

Requiring two probe successes is what stops the breaker flapping.

import time
from dataclasses import dataclass

CLOSED, OPEN, HALF_OPEN = "closed", "open", "half_open"

@dataclass
class Breaker:
    failure_threshold: int = 8         # consecutive failures before opening
    open_seconds: float = 300.0        # rest period before a probe is allowed
    probe_successes: int = 2           # consecutive probe successes needed to close

    state: str = CLOSED
    consecutive_failures: int = 0
    consecutive_probe_successes: int = 0
    opened_at: float = 0.0

    def allow(self, now: float) -> bool:
        if self.state == CLOSED:
            return True
        if self.state == OPEN:
            if now - self.opened_at < self.open_seconds:
                return False
            self.state = HALF_OPEN
            self.consecutive_probe_successes = 0
        return True                     # half-open: a probe is permitted

    def record(self, ok: bool, now: float) -> None:
        if self.state == HALF_OPEN:
            if ok:
                self.consecutive_probe_successes += 1
                if self.consecutive_probe_successes >= self.probe_successes:
                    self.state, self.consecutive_failures = CLOSED, 0
            else:
                self.state, self.opened_at = OPEN, now
            return
        if ok:
            self.consecutive_failures = 0
            return
        self.consecutive_failures += 1
        if self.consecutive_failures >= self.failure_threshold:
            self.state, self.opened_at, self.consecutive_failures = OPEN, now, 0

Requiring two consecutive probe successes before closing is deliberate. A single success proves very little — a load balancer with one healthy backend out of ten will produce intermittent successes indefinitely — and a breaker that closes on the first one will flap, reopening within seconds and generating exactly the churn it exists to prevent.

2. Feed it classified outcomes, not raw statuses #

A breaker driven by “any non-200” trips on 404s, which are a normal part of crawling and say nothing about host health. Drive it from the same classifier the retry layer uses.

BREAKER_FAILURE = {"transient"}                 # connection errors, timeouts, 5xx
BREAKER_NEUTRAL = {"permanent", "throttled"}    # 404 is fine; 429 has its own control
BREAKER_HARD_OPEN = {"refused"}                 # a named 403 opens immediately

def on_outcome(breaker: Breaker, outcome: str, now: float) -> None:
    if outcome in BREAKER_HARD_OPEN:
        breaker.state, breaker.opened_at = OPEN, now
        return
    if outcome in BREAKER_NEUTRAL:
        return
    breaker.record(outcome == "ok", now)

Two distinctions matter here. A 429 is excluded because throttling has its own, gentler control — the rate limiter reduces pace, and opening a circuit on a host that is merely asking you to slow down would be an over-reaction. A refusal that names the crawler opens the circuit immediately with no threshold, because that is a message rather than a fault and no number of further attempts is appropriate.

A per-process breaker in a fleet of ten workers permits ten times the configured failure threshold before anything opens. Move the state to the shared store, mutating it atomically.

-- breaker.lua — KEYS[1]=state hash, ARGV: now, threshold, open_seconds, probe_successes, ok
local st = redis.call('HMGET', KEYS[1], 'state', 'failures', 'opened_at', 'probes')
local state     = st[1] or 'closed'
local failures  = tonumber(st[2]) or 0
local opened_at = tonumber(st[3]) or 0
local probes    = tonumber(st[4]) or 0

local now, threshold      = tonumber(ARGV[1]), tonumber(ARGV[2])
local open_secs, need     = tonumber(ARGV[3]), tonumber(ARGV[4])
local ok                  = ARGV[5] == '1'

if state == 'open' and (now - opened_at) >= open_secs then
  state, probes = 'half_open', 0
end

if state == 'half_open' then
  if ok then
    probes = probes + 1
    if probes >= need then state, failures = 'closed', 0 end
  else
    state, opened_at = 'open', now
  end
elseif ok then
  failures = 0
else
  failures = failures + 1
  if failures >= threshold then state, opened_at, failures = 'open', now, 0 end
end

redis.call('HMSET', KEYS[1], 'state', state, 'failures', failures,
           'opened_at', opened_at, 'probes', probes)
redis.call('EXPIRE', KEYS[1], 86400)
return state

Passing now from the caller keeps the script deterministic and testable, and the day-long expiry means state for hosts no longer being crawled disappears on its own.

One subtlety in the shared case: in half-open, several workers can each be granted a probe simultaneously, which sends a burst at a host that has just been unreachable. Gate the probe with a short-lived lock so only one worker probes at a time.

def try_probe(redis, host: str) -> bool:
    """Only one worker may probe a half-open host at a time."""
    return bool(redis.set(f"breaker:probe:{host}", "1", nx=True, ex=30))

4. Put it in the right place in the request path #

The order is: authorisation check, breaker check, rate token, egress selection, request. Checking the breaker before the rate limiter means an open host never consumes a token, leaving the budget available for hosts that are actually answering — and it means an open circuit costs a single hash lookup rather than a connect timeout.

When the breaker refuses, defer the URL rather than failing it. The scheduler puts it back with a delay matching the breaker’s remaining rest period, and the worker immediately picks up a URL for a different host, which is the politeness-aware frontier behaviour that keeps throughput up.

Verification & Testing #

A breaker driven by raw status codes trips on healthy hosts.

def test_opens_after_threshold_consecutive_failures():
    b, t = Breaker(failure_threshold=3), 0.0
    for _ in range(3):
        assert b.allow(t)
        b.record(False, t)
    assert not b.allow(t)

def test_intermittent_failures_do_not_open():
    b, t = Breaker(failure_threshold=3), 0.0
    for ok in (False, False, True, False, False, True):
        b.allow(t); b.record(ok, t)
    assert b.state == CLOSED

def test_half_open_needs_two_successes():
    b = Breaker(failure_threshold=1, open_seconds=10.0, probe_successes=2)
    b.record(False, 0.0)
    assert not b.allow(5.0)
    assert b.allow(11.0) and b.state == HALF_OPEN
    b.record(True, 11.0)
    assert b.state == HALF_OPEN            # one success is not enough
    b.record(True, 12.0)
    assert b.state == CLOSED

def test_named_refusal_opens_immediately():
    b = Breaker(failure_threshold=99)
    on_outcome(b, "refused", 0.0)
    assert not b.allow(0.0)

The second test is the one that prevents a breaker from being useless: a host with a 30% error rate should keep being crawled, and only a genuine run of failures should open the circuit.

Export breaker state per host as a gauge with three levels. A simultaneous rise in open circuits across unrelated hosts is almost always a problem on your side — DNS, egress, an expired certificate bundle — and that pattern is visible in the gauge long before it is visible in success rates.

Compliance & Operational Guardrails #

A refusal that names the crawler opens the circuit immediately and raises a case for a human.
An open circuit defers URLs; it never routes them through a different egress endpoint to retry.
Breaker state is shared across the fleet so the threshold means what it says.
Probes are single, gated by a lock, and separated by the full rest period.
Every state transition is logged with the outcome that caused it.

Common Mistakes #

Opening on any non-200. A crawl full of 404s opens circuits on healthy hosts and stops collecting.
Closing on a single probe success. The breaker flaps, producing more churn than having none.
Per-process breakers in a fleet. The effective threshold is multiplied by the worker count.

Frequently Asked Questions #

What threshold and rest period should I start with? #

Eight consecutive failures and a five-minute rest is a reasonable default for general crawling. Shorten the rest for hosts you crawl continuously and lengthen it for ones you sweep occasionally — the rest period is essentially a guess at how long an outage lasts, and there is no need for it to be the same everywhere.

How does this interact with the retry budget? #

They compose. The budget bounds retries within the period before the breaker opens; the breaker stops attempts entirely once the host is clearly unavailable. A budget without a breaker keeps generating a trickle of doomed requests indefinitely; a breaker without a budget lets each individual URL burn its full attempt sequence on the way to opening.

Should a manual override exist? #

Yes, in the “force open” direction only. An operator needs to be able to pause a host immediately — after a complaint, a notice, or a mistaken crawl — and that pause should not be overridable by the automatic close. A manual “force closed” is the one to leave out, since its only use is overriding a signal the system is correctly acting on.

Exponential Backoff and Retry Logic — the retry layer this breaker sits above.
Handling 429 Too Many Requests Automatically — why throttling gets a gentler control than failure.
Distributed Crawl Scheduling and Queues — deferring URLs for an open host instead of failing them.