Caching robots.txt With a Stale-While-Revalidate Window #

Fetching a rules file before every request is unthinkable at any real crawl rate; never refetching it is a compliance failure waiting to happen. The workable middle is a cache with an explicit freshness window, a bounded stale window during which a slightly old snapshot may still be used, and a hard stop beyond that. This guide implements that policy, as part of parsing robots.txt programmatically in the Compliance & Ethical Crawling Foundations section.

Problem Framing #

A crawl touching a thousand hosts at ten requests per second would issue ten thousand rules fetches per second without caching — more traffic than the crawl itself, aimed at a file that changes perhaps monthly. So everyone caches. The question is what happens when the cached entry expires while a fetch is in flight, or when the rules endpoint is temporarily unreachable.

Only the last state stops the crawl; the others never block a worker.

Three policies are common and only one is defensible. Cache forever is the fastest and means a site’s new Disallow never takes effect. Fail closed on expiry is correct but brittle: a brief outage of the rules endpoint stops the crawl entirely, which teams then work around by raising the cache lifetime until it is effectively infinite. Stale-while-revalidate with a hard limit keeps crawling on a recent snapshot while a refresh happens in the background, and stops when the snapshot is genuinely too old to rely on.

The last policy needs three numbers per host: a fresh window, during which the snapshot is used without question; a stale window, during which it is used while a refresh is attempted; and a hard age beyond which the host is not crawlable at all.

Step-by-Step Implementation #

1. Model the snapshot and its three ages #

Only the first sight of a host ever blocks on a network fetch.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

FRESH = timedelta(hours=6)
STALE_OK = timedelta(hours=24)      # usable while a refresh is attempted
HARD_MAX = timedelta(days=3)        # beyond this the host is not crawlable

@dataclass(frozen=True)
class RulesSnapshot:
    host: str
    text: str
    sha256: str
    etag: str | None
    last_modified: str | None
    fetched_at: datetime
    status: str                     # "fetched" | "absent" | "unavailable"

    def age(self, now: datetime | None = None) -> timedelta:
        return (now or datetime.now(timezone.utc)) - self.fetched_at

    def state(self, now: datetime | None = None) -> str:
        age = self.age(now)
        if age <= FRESH:
            return "fresh"
        if age <= STALE_OK:
            return "stale_usable"
        if age <= HARD_MAX:
            return "stale_degraded"
        return "expired"

Separating stale_usable from stale_degraded gives somewhere to hang an alert. A host sitting in stale_usable for a few minutes during a refresh is normal; one in stale_degraded for an hour means the refresh path is broken and somebody should know before the snapshot expires entirely.

2. Refresh conditionally #

The rules file is a small, cacheable resource, and both ETag and Last-Modified are widely served for it. A conditional request that returns 304 costs a round trip and no body, which makes frequent revalidation cheap.

import hashlib
import httpx

def refresh(client: httpx.Client, snapshot: RulesSnapshot | None, origin: str) -> RulesSnapshot:
    headers = {}
    if snapshot:
        if snapshot.etag:
            headers["If-None-Match"] = snapshot.etag
        elif snapshot.last_modified:
            headers["If-Modified-Since"] = snapshot.last_modified

    now = datetime.now(timezone.utc)
    try:
        resp = client.get(f"{origin}/robots.txt", headers=headers, timeout=10.0)
    except httpx.TransportError:
        # Unreachable: keep the old snapshot and let the age policy decide.
        return snapshot or RulesSnapshot(origin, "", "", None, None, now, "unavailable")

    if resp.status_code == 304 and snapshot:
        # Unchanged — but the snapshot is now demonstrably current, so reset its age.
        return replace_fetched_at(snapshot, now)
    if resp.status_code == 404:
        return RulesSnapshot(origin, "", "", None, None, now, "absent")
    if resp.status_code >= 400:
        return snapshot or RulesSnapshot(origin, "", "", None, None, now, "unavailable")

    text = resp.text
    return RulesSnapshot(
        host=origin,
        text=text,
        sha256=hashlib.sha256(text.encode("utf-8")).hexdigest(),
        etag=resp.headers.get("ETag"),
        last_modified=resp.headers.get("Last-Modified"),
        fetched_at=now,
        status="fetched",
    )

Resetting fetched_at on a 304 is the detail that makes conditional requests worthwhile. Without it, an unchanged file ages out of the fresh window every six hours and is re-downloaded in full; with it, a 304 is as good as a fetch for freshness purposes at a fraction of the cost.

3. Decide per request, without blocking the fetch #

def rules_for(host: str, cache, refresher) -> tuple[RulesSnapshot, str]:
    """Return (snapshot, disposition). Never blocks on a network fetch."""
    snapshot = cache.get(host)
    if snapshot is None:
        snapshot = refresher.refresh_now(host)        # first sight of a host: block once
        cache.put(host, snapshot)
        return snapshot, "fetched"

    state = snapshot.state()
    if state == "fresh":
        return snapshot, "cached"
    if state in ("stale_usable", "stale_degraded"):
        refresher.schedule(host)                       # background refresh, do not wait
        return snapshot, state
    raise RulesExpired(f"{host}: rules snapshot is {snapshot.age()} old; host not crawlable")

Blocking exactly once, on first sight of a host, is the right trade. That fetch has to happen before any request to the host, and it is a single round trip amortised over the whole crawl of that host. Every subsequent expiry is handled in the background, so no worker ever waits on a rules refresh.

4. Coalesce concurrent refreshes #

Twenty workers noticing the same expiry simultaneously must not produce twenty fetches. A per-host refresh lock — in-process for a single worker, in the shared store for a fleet — collapses them into one.

def schedule(self, host: str) -> None:
    # SET NX with a short TTL: only one worker wins the right to refresh.
    if self.redis.set(f"rules:refreshing:{host}", "1", nx=True, ex=30):
        self.pool.submit(self._refresh_and_store, host)

The lock TTL should exceed the fetch timeout so a crashed refresher does not hold it, and be short enough that a failed refresh is retried promptly.

Verification & Testing #

Time is the awkward dependency, so inject it. Pass now explicitly into the state calculation and the tests become trivial and deterministic:

A 304 keeps the snapshot current for a fraction of a full fetch.

def test_state_transitions():
    base = datetime(2026, 1, 1, tzinfo=timezone.utc)
    snap = RulesSnapshot("h", "", "", None, None, base, "fetched")
    assert snap.state(base + timedelta(hours=1))  == "fresh"
    assert snap.state(base + timedelta(hours=12)) == "stale_usable"
    assert snap.state(base + timedelta(days=2))   == "stale_degraded"
    assert snap.state(base + timedelta(days=4))   == "expired"

def test_304_resets_age(mock_transport):
    mock_transport.script = [(304, "", {})]
    refreshed = refresh(client, snapshot_two_days_old, "https://example.com")
    assert refreshed.sha256 == snapshot_two_days_old.sha256
    assert refreshed.state() == "fresh"

def test_expired_snapshot_blocks_the_host(cache):
    cache.put("example.com", snapshot_five_days_old)
    with pytest.raises(RulesExpired):
        rules_for("example.com", cache, refresher)

In production, export the snapshot age per host as a gauge and the disposition as a counter. The ratio of cached to fetched tells you whether the windows are sized sensibly, and any sustained population in stale_degraded is a refresh path that has stopped working.

Compliance & Operational Guardrails #

A host with no snapshot is fetched before its first request, never crawled optimistically.
An expired snapshot stops the host; it is never extended to keep a run going.
Every crawl event records the snapshot hash and its disposition, so the rules in force are provable.
A 404 is recorded as “no rules published” and is distinct from “rules unavailable”.
Refreshes are rate-limited like any other request to that host.

Common Mistakes #

Raising the cache lifetime to work around a flaky refresh. It converts an operational problem into a compliance one and hides the original fault.
Treating a 5xx on the rules endpoint as a 404. One means “cannot serve rules right now”, the other means “no rules exist”; conflating them is failing open.
Refreshing without coalescing. A synchronised expiry across a fleet produces a burst of identical requests at exactly the moment the host may already be struggling.

Frequently Asked Questions #

What are sensible window values? #

Six hours fresh, twenty-four hours usable, three days hard maximum is a reasonable default for general crawling. Shorten all three for hosts under active negotiation or with a history of policy changes, and record the values in the host’s registry entry so they are visible rather than buried in a constant.

Should the cache be shared across workers? #

Yes, for the same reason the rate limiter is: the snapshot is a property of the host, not of the process. A shared cache also makes refresh coalescing possible and gives one place to inspect what every worker currently believes about a host.

What happens on a redirect from the rules URL? #

Follow it within the same host and scheme, and treat a cross-host redirect as “rules unavailable” rather than adopting another site’s file. A rules file is scoped to an origin, and a redirect to a CDN error page or a marketing site is far more common than a legitimate cross-origin rules document.

Should the cache be keyed by origin or by host? #

By origin — scheme, host and port together. A rules file served over HTTPS does not govern a plain HTTP origin on the same host, and a service on a non-default port publishes its own file. Collapsing them is usually harmless and occasionally wrong in a way that is difficult to notice, since the resulting decisions look plausible. Keying by origin costs nothing and removes the ambiguity entirely.

What should happen when a redirect chain leads somewhere unexpected? #

Follow redirects within the same origin, and treat a redirect that leaves it as “rules unavailable” rather than adopting the destination’s file. Cross-origin redirects on a rules path are far more often a misconfigured CDN, a captive portal, or a marketing landing page than a legitimate delegation. Recording the destination in the snapshot’s status makes the pattern visible: a host whose rules path has redirected to an unrelated origin for a month is worth raising with its operator, and in the meantime the conservative reading applies.

Parsing robots.txt Programmatically — the parser and precedence rules the cached snapshot feeds.
Testing robots.txt Rules in CI — asserting on the snapshots this cache produces.
Conditional Requests With ETag and Last-Modified — the same revalidation mechanics applied to content pages.