Conditional Requests With ETag and Last-Modified #

A conditional request asks the server “has this changed since the version I hold?” and, when the answer is no, costs a round trip and roughly two hundred bytes instead of a full document. It is the cheapest politeness measure available to a crawler and one of the most commonly implemented incorrectly. This guide covers the mechanics and the failure modes, as part of incremental crawling and change detection in the Pipeline Storage, Deduplication & Monitoring section.

Problem Framing #

Two validators exist and they behave differently. An ETag is an opaque token the server associates with a specific version of a resource; the client returns it in If-None-Match. A Last-Modified date is a timestamp with one-second resolution; the client returns it in If-Modified-Since.

Weak validators are entirely adequate for change detection.

ETags are stronger — they detect any change, including two edits within the same second — but they come in two flavours. A strong ETag ("abc123") means byte-identical; a weak one (W/"abc123") means semantically equivalent, which is what a CDN emits when it may serve differently compressed variants. For a crawler, weak validators are perfectly adequate and often preferable.

The failure modes are specific: storing the ETag without its quotes so the comparison never matches; sending If-Modified-Since with a timestamp you generated rather than the one the server sent; treating a 304 as an error; and failing to notice that a site emits a fresh ETag on every response, making the whole exercise pointless.

Step-by-Step Implementation #

1. Store the validators exactly as received #

A 304 may carry fresh headers; merging them keeps the state current.

from dataclasses import dataclass

@dataclass
class Validators:
    etag: str | None = None            # stored WITH quotes and any W/ prefix
    last_modified: str | None = None   # stored as the server's exact date string

    def conditional_headers(self) -> dict[str, str]:
        """ETag takes precedence; sending both is unnecessary."""
        if self.etag:
            return {"If-None-Match": self.etag}
        if self.last_modified:
            return {"If-Modified-Since": self.last_modified}
        return {}

    @classmethod
    def from_response(cls, response, previous: "Validators | None" = None) -> "Validators":
        return cls(
            etag=response.headers.get("ETag") or (previous.etag if previous else None),
            last_modified=response.headers.get("Last-Modified")
                          or (previous.last_modified if previous else None),
        )

Storing the ETag verbatim — quotes, W/ prefix and all — is the single most common bug in this area. The value is opaque and must be echoed exactly; stripping the quotes for tidiness means the server never recognises it and every request transfers a full body while appearing to be conditional.

Reformatting the Last-Modified date is the same mistake in a different costume. Store the string the server sent; do not parse it to a datetime and render it back, because the reformatted version may differ in a way that defeats the comparison.

2. Handle the three response classes #

def conditional_get(client, url: str, validators: Validators):
    response = client.get(url, headers=validators.conditional_headers())

    if response.status_code == 304:
        # Not an error. The server may refresh validators even on a 304.
        return "not_modified", None, Validators.from_response(response, validators)

    if response.status_code == 200:
        return "modified", response, Validators.from_response(response)

    if response.status_code in (404, 410):
        return "gone", None, Validators()

    response.raise_for_status()
    return "error", response, validators

A 304 carries no body and may carry updated headers — including a refreshed ETag or Cache-Control. Merging rather than discarding them keeps the stored state current.

Note that a 304 does not mean the resource is identical to what you parsed; it means it is identical to the version the validator identifies. If your stored record was produced by a different extractor version, a 304 still means the page is unchanged, and re-extraction requires a deliberate refetch rather than a conditional one.

3. Detect a host whose validators are useless #

Some sites emit a new ETag on every response — generated from a timestamp, a server identifier, or the response’s own compression. Conditional requests against them never produce a 304, and the crawl pays the full transfer while believing it is being efficient.

from collections import defaultdict

class ValidatorEfficacy:
    """Track whether conditional requests actually produce 304s per host."""

    def __init__(self, sample: int = 200):
        self.sample = sample
        self.conditional = defaultdict(int)
        self.not_modified = defaultdict(int)

    def observe(self, host: str, was_conditional: bool, was_304: bool) -> None:
        if was_conditional:
            self.conditional[host] += 1
            if was_304:
                self.not_modified[host] += 1

    def is_useless(self, host: str) -> bool:
        attempts = self.conditional[host]
        return attempts >= self.sample and self.not_modified[host] == 0

When a host is flagged useless, stop sending conditional headers to it — they add a header and buy nothing — and rely on the fingerprint layer instead, which still removes the parse and write. Record the finding per host so it is not rediscovered every quarter.

4. Account for a 304 in the rate budget #

A conditional request is still a request. It occupies a connection, reaches the origin or its cache, and counts against whatever budget the host has set.

def fetch_incremental(client, state, limiter, budget) -> tuple[str, object | None]:
    wait = limiter.acquire(state.host)          # a 304 costs a token like anything else
    if wait:
        return "deferred", None
    outcome, response, validators = conditional_get(client, state.url, state.validators)
    state.validators = validators
    budget.charge(state.host, bytes_transferred=len(response.content) if response else 0)
    return outcome, response

What conditional requests reduce is bandwidth, not request count. Reducing the request count is the job of the revisit scheduler, and conflating the two leads to a crawler that politely transfers very little while still hitting a host far more often than it needs to.

Verification & Testing #

A host that never returns 304 makes the header pure overhead.

def test_etag_is_echoed_verbatim(mock_transport):
    mock_transport.script = [(200, "body", {"ETag": 'W/"abc-123"'}), (304, "", {})]
    validators = Validators()
    _, response, validators = conditional_get(client, URL, validators)
    assert validators.etag == 'W/"abc-123"'          # quotes and prefix intact
    outcome, _, _ = conditional_get(client, URL, validators)
    assert outcome == "not_modified"
    assert mock_transport.last_request.headers["If-None-Match"] == 'W/"abc-123"'

def test_last_modified_used_when_no_etag(mock_transport):
    date = "Wed, 14 Jun 2026 09:30:00 GMT"
    mock_transport.script = [(200, "body", {"Last-Modified": date}), (304, "", {})]
    _, _, validators = conditional_get(client, URL, Validators())
    conditional_get(client, URL, validators)
    assert mock_transport.last_request.headers["If-Modified-Since"] == date
    assert "If-None-Match" not in mock_transport.last_request.headers

def test_304_refreshes_headers(mock_transport):
    mock_transport.script = [(304, "", {"ETag": '"new"', "Cache-Control": "max-age=600"})]
    _, _, validators = conditional_get(client, URL, Validators(etag='"old"'))
    assert validators.etag == '"new"'

def test_useless_validator_detection():
    efficacy = ValidatorEfficacy(sample=10)
    for _ in range(10):
        efficacy.observe("noisy.example", was_conditional=True, was_304=False)
    assert efficacy.is_useless("noisy.example")

In production, export the 304 rate per host. It is one of the most informative single numbers about a crawl: a high rate means the incremental machinery is working, a rate of zero on a host you check frequently means either the content genuinely changes every time or the validators are useless, and the efficacy detector distinguishes the two.

Compliance & Operational Guardrails #

A 304 is charged to the host’s rate budget and recorded in the audit trail like any other fetch.
Validators are stored verbatim and never normalised or reformatted.
Conditional headers are dropped for hosts where they demonstrably never produce a 304.
A 404 or 410 clears the stored validators and marks the record as gone.
Revisit intervals still honour the host’s published crawl delay, whatever the 304 rate.

Common Mistakes #

Stripping quotes from the ETag. The comparison never matches and every request transfers a full body.
Regenerating the If-Modified-Since date. A reformatted timestamp may not match what the server expects.
Treating 304 as an error. The retry layer refetches, and the saving is inverted into extra load.

Frequently Asked Questions #

Should both validators be sent together? #

No need. If-None-Match takes precedence where both are present, so If-Modified-Since is ignored. Sending both is harmless but signals uncertainty; send the ETag when you have one and fall back to the date when you do not.

What about `Cache-Control` and `Expires`? #

Worth reading and worth honouring as a lower bound on the revisit interval. A resource declaring max-age=86400 is telling you it does not expect to change within a day, and rechecking it hourly is wasted effort on both sides. Treat it as advice about frequency rather than as a guarantee about content.

Do conditional requests work through a proxy pool? #

Yes, but be aware that a shared cache in front of the origin may answer differently depending on which endpoint the request leaves from. Host affinity — a stable, small egress subset per host — keeps that consistent, which is one more reason for the affinity-based selection described in the network section.

What should happen when a host returns a `200` with identical content? #

Fall through to the fingerprint layer, which will recognise the record as unchanged and skip the downstream work. That combination — a full transfer that turns out to be unchanged — is worth counting per host, because a persistently high rate means the host’s validators are not working and the transfer is pure waste. Where the volume justifies it, that is a reasonable thing to raise with the operator; a site sending fresh validators on unchanged content is paying for the bandwidth too.

Do conditional requests interact with compression? #

Only in that a weak ETag frequently exists because the origin serves several compressed variants. Weak validators are entirely adequate for change detection — semantic equivalence is what a crawler cares about — so accept them rather than treating them as second class. What matters is echoing the validator verbatim, including the W/ prefix.

Incremental Crawling and Change Detection — the three layers this one sits in.
Detecting Content Changes With Rolling Hashes — what to do once a body has been transferred.
Caching robots.txt With a Stale-While-Revalidate Window — the same mechanics applied to the rules file.