Compliant Web Scraping

Adaptive Rate Limiting Driven by Response Headers #

Servers tell you how fast you may go — if you read the right headers. This page shows how to steer a token-bucket limiter directly from Retry-After, the draft IETF RateLimit-Limit/RateLimit-Remaining/RateLimit-Reset fields, and the 429/503 status codes that signal you have already overshot. It is a detail page under implementing polite rate limiting in the Compliance & Ethical Crawling Foundations section, and it assumes you already run a fixed-rate limiter and now want it to react to what the server actually reports instead of guessing at a safe constant.

Problem Framing #

A static rate is always wrong in one of two directions. Set it conservatively and you leave throughput on the table during quiet hours; set it aggressively and you trip defensive limits the moment traffic spikes on the server side. The information you need to size the rate correctly is already on the wire — modern APIs return their current quota and reset window in response headers on every successful call, and they return an explicit wait instruction when they reject you.

Adaptive limiting closes that loop. On each response you read the server’s own accounting, translate it into a refill rate and a bucket capacity, and adjust before the next request rather than after a ban. The hard part is that the header vocabulary is not fully standardised: Retry-After is stable (RFC 9110), but the RateLimit-* family is an IETF draft that servers implement with variations — sometimes a remaining count, sometimes a reset as delta-seconds, sometimes as a Unix timestamp. A robust client reads defensively and degrades gracefully when a header is absent or malformed.

Step-by-Step Implementation #

  1. Run a token bucket as the base limiter. Capacity is your burst allowance; the refill rate is tokens per second. Every request costs one token and blocks until one is available.
  2. On 429 or 503, honour Retry-After verbatim. Parse it as delta-seconds or an HTTP-date, drain the bucket, and refuse to issue tokens until the wait elapses. This directive overrides any computed rate.
  3. On success, read RateLimit-Remaining and RateLimit-Reset and recompute the refill rate as remaining / seconds_until_reset, so you glide into the reset boundary instead of exhausting the quota early.
  4. Clamp to a polite ceiling and floor so a generous server cannot push you above your own baseline and a hostile one cannot stall you indefinitely.
  5. Treat missing headers as “hold current rate” — never accelerate on the absence of a signal.
import time, email.utils

class AdaptiveTokenBucket:
    def __init__(self, rate: float, capacity: float, max_rate: float):
        self.rate, self.capacity, self.max_rate = rate, capacity, max_rate
        self.tokens = capacity
        self.updated = time.monotonic()
        self.blocked_until = 0.0

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now

    def acquire(self):
        while True:
            now = time.monotonic()
            if now < self.blocked_until:          # a Retry-After hold is active
                time.sleep(self.blocked_until - now)
                continue
            self._refill()
            if self.tokens >= 1:
                self.tokens -= 1
                return
            time.sleep((1 - self.tokens) / self.rate)

    def observe(self, status: int, headers: dict):
        # 429/503 + Retry-After: an explicit, authoritative wait. Honour it.
        if status in (429, 503) and "Retry-After" in headers:
            self.blocked_until = time.monotonic() + _parse_retry_after(headers["Retry-After"])
            self.tokens = 0
            return
        # Draft IETF RateLimit-* headers: recompute refill from server accounting.
        rem, reset = headers.get("RateLimit-Remaining"), headers.get("RateLimit-Reset")
        if rem is not None and reset is not None:
            secs = max(float(reset), 1.0)          # reset as delta-seconds
            new_rate = float(rem) / secs
            self.rate = min(max(new_rate, 0.1), self.max_rate)  # clamp floor/ceiling

def _parse_retry_after(value: str) -> float:
    value = value.strip()
    if value.isdigit():
        return float(value)                        # delta-seconds form
    dt = email.utils.parsedate_to_datetime(value)  # HTTP-date form
    return max(0.0, dt.timestamp() - time.time())

Wire it into the request loop so every response feeds the limiter:

bucket = AdaptiveTokenBucket(rate=2.0, capacity=5, max_rate=10.0)

def fetch(session, url):
    bucket.acquire()                               # blocks until a token is free
    resp = session.get(url, timeout=10)
    bucket.observe(resp.status_code, resp.headers) # adapt from what the server reported
    return resp

When the server sends no useful headers and only repeated 429s, fall back to exponential backoff and retry logic so you still escalate your wait safely.

Verification & Testing #

Drive the limiter with a mock response instead of a live server so the header handling is deterministic:

def test_retry_after_blocks_the_bucket():
    b = AdaptiveTokenBucket(rate=5.0, capacity=5, max_rate=10.0)
    b.observe(429, {"Retry-After": "2"})
    assert b.tokens == 0
    assert b.blocked_until > time.monotonic()

def test_remaining_lowers_the_rate():
    b = AdaptiveTokenBucket(rate=10.0, capacity=10, max_rate=10.0)
    b.observe(200, {"RateLimit-Remaining": "5", "RateLimit-Reset": "50"})
    assert b.rate == 0.1   # 5 requests over 50s, clamped to the floor

Against a real endpoint, httpbin can force the reject path so you can watch the hold engage:

# Returns 429 with a Retry-After header your observe() call should honour.
curl -sD - -o /dev/null "https://httpbin.org/status/429"

In production, export the current bucket.rate as a gauge and alert if it sits at the floor for a sustained window — that indicates the server is throttling you continuously and the crawl needs manual review.

Compliance & Operational Guardrails #

  • Retry-After is a server instruction, not a hint. Overriding it with a shorter computed delay is the clearest possible evidence of impolite crawling; always honour it verbatim, plus a small safety margin.
  • Never let a generous quota override your own ceiling. A server advertising thousands of requests per minute does not obligate you to use them — cap at a max_rate that reflects the site’s likely capacity and your Terms of Service commitments.
  • Log every rate adjustment with its trigger (status code, Retry-After, or RateLimit-* recompute) so your audit trail can demonstrate the crawler paced itself in response to the server’s own signals.

Common Mistakes #

  1. Accelerating when headers are missing. Absence of a RateLimit-Remaining field is not a green light; hold your current rate rather than ramping up on no evidence.
  2. Parsing Retry-After as seconds only. RFC 9110 permits an HTTP-date form; a client that assumes an integer will wait zero seconds on a date-valued header and hammer the server immediately.
  3. Treating RateLimit-Reset inconsistently. Some servers send delta-seconds, others a Unix timestamp; detect which — a value far larger than any plausible window is an absolute timestamp — before dividing, or your computed rate will be wildly wrong.

Frequently Asked Questions #

Does Retry-After override my computed backoff delay? #

Yes. When a server sends Retry-After on a 429 or 503, honour it verbatim and suspend token issuance until it elapses, regardless of what your token bucket or backoff formula would otherwise permit. It is an explicit, authoritative instruction; a shorter self-computed delay both risks a ban and undermines any good-faith compliance claim.

The RateLimit-* headers are only an IETF draft — are they safe to rely on? #

Rely on them opportunistically, not exclusively. Read them when present to tune your rate more precisely, but because implementations vary in units and naming, always keep Retry-After plus 429/503 handling as the authoritative fallback. Parse the draft headers defensively and ignore values you cannot interpret rather than acting on a misread.

How does this interact with concurrency across multiple workers? #

A single in-process bucket only paces one worker. For a distributed crawl, back the token state with a shared store such as Redis so every worker draws from the same quota, and have whichever worker sees a Retry-After write the hold centrally so the whole fleet pauses together rather than each retrying independently.