Incremental Crawling and Change Detection #

A crawl that refetches everything on every pass spends most of its budget confirming that nothing changed. An incremental crawl spends it where the content actually moved — which is better for your throughput, dramatically better for the hosts you visit, and the only way to answer “what changed since last week” at all. This topic, part of the Pipeline Storage, Deduplication & Monitoring section, covers the mechanisms that make a crawl incremental: conditional requests, change fingerprints, and a revisit policy that learns each page’s own rhythm.

Core Principles: Three Independent Layers #

Incremental crawling is often reduced to “use ETags”, which is one third of the picture. Three layers each remove a different portion of the wasted work, and they compose.

Budget freed from static pages goes to the pages that actually move.

Conditional requests remove the transfer. A 304 Not Modified costs a round trip and no body — typically a 98% reduction in bytes for an unchanged page — but it still costs a request against the host’s budget.

Change fingerprints remove the downstream work. When a page has changed at the byte level but not in the fields you extract — a rotating advertisement, a view counter, a timestamp in the footer — the fetch was necessary and the parse, validation and write were not.

Revisit scheduling removes the request. A page that has not changed in six months does not need checking daily, and one that changes hourly should not be checked weekly. Learning the rate per URL is what turns a fixed-interval crawl into one that spends its budget where change actually happens.

Applied together on a typical catalogue crawl, these reduce bytes by an order of magnitude and requests by more than half, while improving freshness on the pages that move — because the budget freed from static pages goes to volatile ones.

What “Changed” Means Is a Decision #

The most consequential design choice is what counts as a change. Byte-level equality is the wrong answer for almost every crawl: a page carrying a session identifier, a rendered timestamp or a rotating promotional slot differs on every fetch while conveying identical information.

The right definition is change in the extracted record. A page whose price, title, availability and description are unchanged has not changed, whatever the markup did. This has a pleasing consequence: the fingerprint is computed from data you already produce, so it costs one hash per record and no additional parsing.

import hashlib
import json

CHANGE_FIELDS = ("title", "price_minor", "currency", "availability", "description")

def change_fingerprint(record: dict) -> str:
    """Hash of the fields whose change is meaningful — not the raw response."""
    payload = {field: normalise(record.get(field)) for field in CHANGE_FIELDS}
    encoded = json.dumps(payload, sort_keys=True, ensure_ascii=False,
                         separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()

Keeping the field list explicit and versioned matters. Adding a field to the fingerprint makes every page appear changed on the next pass, which is a legitimate one-off cost but needs to be a decision rather than a surprise — the same versioning discipline the content hashing guide applies to deduplication keys.

Technical Implementation #

Storing the crawl state per URL #

Three outcomes, tracked separately, because they mean different things.

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class UrlState:
    url: str
    etag: str | None = None
    last_modified: str | None = None
    fingerprint: str | None = None
    last_fetched_at: datetime | None = None
    last_changed_at: datetime | None = None
    unchanged_streak: int = 0
    check_interval: timedelta = timedelta(days=1)
    next_check_at: datetime | None = None

    def observe_unchanged(self, now: datetime, max_interval: timedelta) -> None:
        self.last_fetched_at = now
        self.unchanged_streak += 1
        # Back off geometrically, bounded.
        self.check_interval = min(max_interval, self.check_interval * 2)
        self.next_check_at = now + self.check_interval

    def observe_changed(self, now: datetime, fingerprint: str,
                        min_interval: timedelta) -> None:
        self.last_fetched_at = now
        self.last_changed_at = now
        self.fingerprint = fingerprint
        self.unchanged_streak = 0
        # A change means we were checking too slowly: halve, bounded below.
        self.check_interval = max(min_interval, self.check_interval / 2)
        self.next_check_at = now + self.check_interval

The multiplicative backoff and halving on change is the same shape as an adaptive rate limiter, and it converges usefully: a page changing daily settles near a daily check, a static page drifts out to the maximum interval, and both adapt within a few observations when their behaviour changes.

The fetch path #

def incremental_fetch(client, state: UrlState, now: datetime, config) -> tuple[str, dict | None]:
    """Return (outcome, record). Outcome is 'not_modified' | 'unchanged' | 'changed'."""
    headers = {}
    if state.etag:
        headers["If-None-Match"] = state.etag
    elif state.last_modified:
        headers["If-Modified-Since"] = state.last_modified

    response = client.get(state.url, headers=headers)

    if response.status_code == 304:
        state.observe_unchanged(now, config.max_interval)
        metrics.incremental.labels(outcome="not_modified").inc()
        return "not_modified", None

    state.etag = response.headers.get("ETag") or state.etag
    state.last_modified = response.headers.get("Last-Modified") or state.last_modified

    record = parse_and_validate(response)
    fingerprint = change_fingerprint(record)

    if fingerprint == state.fingerprint:
        state.observe_unchanged(now, config.max_interval)
        metrics.incremental.labels(outcome="unchanged").inc()
        return "unchanged", None                  # nothing downstream needs to run

    state.observe_changed(now, fingerprint, config.min_interval)
    metrics.incremental.labels(outcome="changed").inc()
    return "changed", record

The three outcomes should be tracked separately because they mean different things. A high not_modified rate means the host implements conditional requests well and you are saving transfer. A high unchanged rate means you are fetching bodies for nothing — the host is not sending validators, and it is worth checking whether it would. A low changed rate on pages you check frequently means the interval is too short.

Seeding the interval #

A new URL has no history, so its first interval is a guess. Guess by page type rather than globally: a listing index changes far more often than an archived article.

SEED_INTERVALS = {
    "index":   timedelta(hours=6),
    "listing": timedelta(days=1),
    "detail":  timedelta(days=3),
    "archive": timedelta(days=30),
}

Recording the seed and letting the adaptive logic take over from the second observation gives sensible behaviour immediately and correct behaviour within a week.

Compliance Boundaries #

Incremental crawling is a politeness measure as much as an efficiency one, and it interacts with several obligations.

Conditional requests reduce bandwidth, not request count.

Conditional requests still count against the host's rate budget
A 304 A `304` is recorded in the audit trail like any other fetch
The rules file and terms are re-checked on the normal schedule regardless of content stability
Change history retains only what the retention class permits — a per-change record of personal data is still personal data
A page that disappears is handled explicitly rather than left as a stale record
Revisit intervals never fall below the host's published crawl delay

The fourth point is easy to miss. Storing a change history — every observed version of a record — multiplies the volume of personal data you hold and extends its lifetime, because the oldest version is as old as the first crawl. Where change history is genuinely needed, give it its own, usually shorter, retention class and exclude personal fields from the historical rows.

The fifth is the one most incremental crawlers get wrong. A 404 or 410 on a previously-present URL is a change, and the correct handling is to mark the record as gone rather than to leave the last successful version in the dataset indefinitely. A dataset that never removes anything drifts steadily away from reality.

Scheduling Revisits From Observed Behaviour #

The third layer — deciding when to check at all — is where most of the saving lives, and it is the layer most crawls never implement. A fixed interval treats a price that changes hourly and an archived article identically, which means it is simultaneously too slow for one and wasteful for the other.

The adaptive interval described above converges usefully, but two refinements make it substantially better.

Priority weighting. Not all pages are equally worth checking. A page feeding a live product or a monitored alert deserves a shorter floor than one feeding a quarterly report, regardless of how often it changes. Multiply the computed interval by a per-page-type weight so importance and volatility both influence the schedule.

Budget-aware scheduling. A host has a finite request budget, and the revisit scheduler is competing with discovery for it. Rather than letting every due URL enter the frontier, rank the due set by expected value — how likely the page is to have changed, times how much that change matters — and admit only what the budget supports.

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class RevisitCandidate:
    url: str
    host: str
    due_at: datetime
    interval: timedelta
    unchanged_streak: int
    priority: float             # 1.0 = normal, higher = more important

    def change_likelihood(self, now: datetime) -> float:
        """Rough hazard estimate: overdue pages are likelier to have changed."""
        if self.interval.total_seconds() <= 0:
            return 1.0
        overdue = (now - self.due_at).total_seconds() / self.interval.total_seconds()
        base = 1.0 / (1.0 + self.unchanged_streak)      # long-stable pages change less
        return min(1.0, base * (1.0 + max(0.0, overdue)))

    def score(self, now: datetime) -> float:
        return self.change_likelihood(now) * self.priority

def select_revisits(candidates: list[RevisitCandidate], now: datetime,
                    budget_per_host: dict[str, int]) -> list[RevisitCandidate]:
    """Spend each host's revisit budget on the URLs most likely to have changed."""
    ordered = sorted((c for c in candidates if c.due_at <= now),
                     key=lambda c: c.score(now), reverse=True)
    remaining = dict(budget_per_host)
    chosen = []
    for candidate in ordered:
        if remaining.get(candidate.host, 0) <= 0:
            continue
        remaining[candidate.host] -= 1
        chosen.append(candidate)
    return chosen

The unchanged_streak term is doing the important work. A page that has been checked twelve times without changing is unlikely to have changed on the thirteenth, and deprioritising it frees budget for pages with a shorter stable history. That single term typically shifts a large fraction of a crawl’s requests from static archive pages onto the pages that actually move.

Note what this does not do: it never exceeds a host’s budget in order to check something important. Priority reorders the queue; it does not expand it. A crawl that raises its rate because a page matters to the crawler is imposing its own priorities on somebody else’s infrastructure, which is precisely the reasoning the polite rate limiting layer exists to prevent.

Seeding and Recovering #

Two edge cases need explicit handling. A new URL has no history, so its first interval comes from a per-page-type seed and adapts from the second observation. A long-dormant URL — one whose interval has drifted out to the maximum — needs an occasional forced check even when its score is low, or a page that starts changing after a year of stability will not be noticed for another year.

FORCE_CHECK_AFTER = timedelta(days=30)

def must_check(candidate: RevisitCandidate, now: datetime,
               last_fetched: datetime) -> bool:
    """A floor on staleness, independent of score, so nothing is forgotten."""
    return now - last_fetched > FORCE_CHECK_AFTER

A hard staleness floor costs very little — it affects only the pages the scoring has deprioritised furthest — and it removes the failure mode where an adaptive scheduler quietly stops covering part of the corpus.

Measuring Whether Any of It Is Working #

Four numbers describe an incremental crawl, and together they say whether each layer is earning its place.

Bytes saved — the share of fetches answered with a 304 — measures the conditional layer. Low on a host that supports validators means they are being sent incorrectly; low across the board means most targets do not send them.

Downstream work saved — the share of transferred bodies whose fingerprint was unchanged — measures the fingerprint layer. A high number here alongside a low 304 rate is the classic signature of a host that transfers unchanged content, which is worth raising with its operator.

Requests saved — the ratio of due checks to total URLs — measures the scheduler. It should rise over the first few weeks as intervals converge and then stabilise.

Freshness — the median age of the most recent observation for pages that did change — is the number that keeps the other three honest. Savings that come at the cost of stale data are not savings; a crawl that halves its requests while doubling the time to notice a price change has made a trade, not an improvement, and the trade should be visible.

Track all four per host and per page type. A change in any one of them usually explains a change in the others, and the combination distinguishes “the site changed” from “our scheduling drifted” far faster than any single metric.

Common Mistakes #

Fingerprinting the raw response. Every rotating advert and timestamp registers as a change, and the fingerprint saves nothing.
Ignoring Last-Modified when no ETag is present. A meaningful share of sites send only the date validator.
Sending both validators at once. If-None-Match takes precedence, so If-Modified-Since is ignored; sending both is harmless but suggests the logic is guessing.
Fixed revisit intervals. Static pages get checked pointlessly while volatile ones go stale.
Treating a disappearance as an error. A removed page is a change, not a fault, and the record should reflect it.

Frequently Asked Questions #

What if a site sends a new ETag on every request? #

Some sites generate the validator from a timestamp or include a per-response identifier, which makes conditional requests useless. Detect it — an ETag that never produces a 304 over many checks — and fall back to the fingerprint layer, which still saves all the downstream work even though the transfer cost remains. Record the finding per host so nobody re-investigates it.

How should deletions be represented? #

As a state on the record, not as a physical delete: set a last_seen_at and a status of gone, so downstream consumers can distinguish “no longer available” from “never existed”. Physical deletion is reserved for retention expiry and erasure requests, which are different operations with different audit requirements — as covered in data retention and GDPR deletion workflows.

Does incremental crawling interact with deduplication? #

Closely. The change fingerprint and the content hash answer adjacent questions — “did this page change since we last saw it” and “is this record the same as another record” — and they should use the same normalisation so their answers are consistent. Sharing the normalisation function is the simplest way to guarantee that.

How much does this actually save? #

On a catalogue crawl of a few hundred thousand pages, conditional requests typically remove 80–95% of the bytes for unchanged pages, fingerprinting removes the parse and write for a further slice, and adaptive scheduling removes 40–70% of the requests outright once intervals have converged. The freed budget is what lets volatile pages be checked more often without increasing total load.

How should an incremental crawl handle a full re-extraction? #

As a separate mode, not by invalidating the change state. When an extractor changes and every page needs re-parsing, the pages themselves have not changed — so the conditional layer should still be used, and the fingerprints should be recomputed from the newly parsed records rather than treated as mismatches. Marking the run as a re-extraction rather than a normal pass keeps the change-rate metrics interpretable, since otherwise a deployment shows up as every page on the site changing at once.

Does incremental crawling make a dataset harder to reason about? #

Only if the observation times are not recorded. Store first_seen, last_seen and last_changed on every record and the dataset is if anything easier to reason about than one produced by a fixed-interval crawl, because the change history is explicit rather than implied by which crawl a row happened to land in. Consumers can then ask “what did we know, and when” directly, which is the question incremental crawling exists to answer.

What is the first layer to implement if only one is affordable? #

The change fingerprint. It requires no cooperation from the target, works on every host regardless of what validators they send, and removes the majority of the downstream cost — parse, validate, dedup, write — which is usually where the pipeline’s real expense sits. Conditional requests and adaptive scheduling both add more, but each depends on something outside your control, whereas the fingerprint works everywhere from the first crawl.

Conditional Requests With ETag and Last-Modified — implementing the validator layer correctly.
Detecting Content Changes With Rolling Hashes — finding what changed, not just that it did.
Deduplicating Records With Content Hashing — the sibling technique sharing the same normalisation.
Distributed Crawl Scheduling and Queues — where revisit times enter the frontier.