Compliant Web Scraping

Testing robots.txt Rules in CI #

A rules parser is a compliance control, and a compliance control that is not tested is a hope. This guide sets up a continuous-integration suite that pins your parser’s behaviour against a fixture corpus, verifies that the crawl’s configured seed paths are actually permitted, and fails the build when a dependency upgrade quietly changes how a directive resolves. It is part of parsing robots.txt programmatically, within the Compliance & Ethical Crawling Foundations section.

Problem Framing #

Two failure modes motivate this. The first is silent behavioural drift: a library upgrade changes how the wildcard group interacts with a named group, or how a $ anchor treats a query string, and a crawl that was in scope on Friday is out of scope on Monday with no code change and no error. The second is configuration drift: someone adds a seed URL or a discovery pattern that points at a path the target’s rules file disallows, and nothing catches it until the crawl has already fetched a few thousand disallowed pages.

Two kinds of drift a suite catchesOne suite pins the parser; the other pins your own configuration.Two kinds of drift a suite catchesBehavioural driftA library upgrade changes precedenceGroup selection resolves differentlyNo code change, no errorCrawl scope widens silentlyConfiguration driftA new seed points at a blocked pathA prefix is added without reviewCaught before the deployFixed in configuration, not in an incident
One suite pins the parser; the other pins your own configuration.

Both are cheap to prevent and expensive to discover in production. A rules parser has small text inputs and boolean outputs, which makes it unusually well suited to table-driven testing, and the seed-path check is a few seconds of work against cached rules files.

Step-by-Step Implementation #

1. Pin parser behaviour with a fixture table #

Fixture cases where implementations divergeEach row is a behaviour worth pinning permanently in your own suite.Fixture cases where implementations divergeCaseCorrect resultCommon bugSpecific group presentWildcard ignoredRules mergedAllow and Disallow tieAllow winsOrder decidesEmpty Disallow valueNothing blockedEverything blockedDollar anchor plus queryRule does not matchRule matchesByte-order mark first lineGroup parsedGroup skipped
Each row is a behaviour worth pinning permanently in your own suite.

Start with the cases where implementations genuinely differ. Group selection, Allow and Disallow precedence, the empty-value case, and the $ anchor account for nearly every real divergence.

import pytest
from crawler.rules import parse

CASES = [
    # (id, rules text, agent, path, expected)
    ("wildcard-blocks",      "User-agent: *\nDisallow: /private/",              "Bot", "/private/x", False),
    ("wildcard-allows",      "User-agent: *\nDisallow: /private/",              "Bot", "/public/x",  True),
    ("specific-wins",        "User-agent: *\nDisallow: /\n"
                             "User-agent: Bot\nDisallow: /tmp/",                "Bot", "/a",         True),
    ("specific-ignores-star","User-agent: *\nDisallow: /\n"
                             "User-agent: Bot\nDisallow: /tmp/",                "Bot", "/tmp/x",     False),
    ("longest-match",        "User-agent: *\nDisallow: /a/\nAllow: /a/b/",      "Bot", "/a/b/c",     True),
    ("tie-goes-to-allow",    "User-agent: *\nDisallow: /a\nAllow: /a",          "Bot", "/a",         True),
    ("empty-disallow",       "User-agent: *\nDisallow:",                        "Bot", "/anything",  True),
    ("dollar-anchor",        "User-agent: *\nDisallow: /*.pdf$",                "Bot", "/doc.pdf",   False),
    ("dollar-with-query",    "User-agent: *\nDisallow: /*.pdf$",                "Bot", "/doc.pdf?v=2", True),
    ("comments-only",        "# nothing here\n",                                "Bot", "/anything",  True),
    ("crlf-and-bom",         "User-agent: *\r\nDisallow: /x/\r\n",        "Bot", "/x/y",       False),
    ("case-insensitive-ua",  "User-agent: bOt\nDisallow: /q/",                  "Bot", "/q/z",       False),
]

@pytest.mark.parametrize("case_id,text,agent,path,expected",
                         CASES, ids=[c[0] for c in CASES])
def test_rule_resolution(case_id, text, agent, path, expected):
    assert parse(text).can_fetch(agent, path) is expected

The specific-ignores-star and dollar-with-query cases are the two that most often change silently between library versions. The byte-order-mark case catches a parser that treats the first line as malformed and therefore skips the whole group — which fails open, the worst possible direction.

2. Assert the crawl’s own configuration against real rules files #

The second suite is about your configuration rather than your parser. For each host in the crawl registry, load the cached rules snapshot and assert that every configured seed and path prefix is permitted.

import pytest
from crawler.registry import load_registry
from crawler.rules import parse

REGISTRY = load_registry("registry/")

@pytest.mark.parametrize("entry", REGISTRY, ids=lambda e: e.host)
def test_configured_paths_are_permitted(entry, rules_snapshots):
    snapshot = rules_snapshots.get(entry.host)
    if snapshot is None:
        pytest.skip(f"no cached rules snapshot for {entry.host}")

    rules = parse(snapshot.text)
    for prefix in entry.allowed_path_prefixes:
        assert rules.can_fetch(entry.agent_token, prefix), (
            f"{entry.host}: configured prefix {prefix} is disallowed by the rules file "
            f"(snapshot {snapshot.sha256[:12]}, fetched {snapshot.fetched_at})"
        )

Use a cached snapshot rather than fetching in the test. A suite that hits the network is slow, flaky, and — worse — sends requests from continuous-integration infrastructure that the target has never seen before. Refresh the snapshots in a scheduled job that runs outside the build, and let the build assert against whatever the most recent snapshot says.

3. Fail the build when a snapshot goes stale #

A snapshot that has not been refreshed is not evidence about the current rules. Treat staleness as a failure with a clear message rather than a silent skip.

from datetime import datetime, timedelta, timezone

MAX_SNAPSHOT_AGE = timedelta(days=7)

def test_snapshots_are_fresh(rules_snapshots):
    now = datetime.now(timezone.utc)
    stale = {
        host: snap.fetched_at
        for host, snap in rules_snapshots.items()
        if now - snap.fetched_at > MAX_SNAPSHOT_AGE
    }
    assert not stale, f"rules snapshots older than {MAX_SNAPSHOT_AGE.days} days: {stale}"

4. Detect a change in what the rules permit #

The most useful check compares the previous snapshot with the current one in terms of outcomes, not text. A site reformatting its file is noise; a site adding a Disallow that covers a path you crawl is not.

def permission_diff(old_text: str, new_text: str, agent: str, paths: list[str]) -> dict[str, tuple[bool, bool]]:
    """Paths whose permitted/denied status changed between two snapshots."""
    old, new = parse(old_text), parse(new_text)
    changed = {}
    for path in paths:
        before, after = old.can_fetch(agent, path), new.can_fetch(agent, path)
        if before != after:
            changed[path] = (before, after)
    return changed

Run this in the snapshot-refresh job and open a ticket when it returns anything. A path moving from permitted to denied should pause that part of the crawl immediately; one moving from denied to permitted is worth a look too, because it often means the site restructured and your prefixes now point somewhere unintended.

Verification & Testing #

The suite verifies itself in one useful way: introduce a deliberate bug and confirm the tests catch it. Change the group-selection function to merge the wildcard and specific groups, and specific-ignores-star should fail. Change the longest-match comparison to first-match, and longest-match should fail. A suite where no mutation causes a failure is testing nothing.

Where each check runsSnapshots refresh outside the build so no test ever hits the network.Where each check runs1Commit: parserfixture table2Commit: configuredprefixes3Nightly: refreshsnapshots4Nightly: diffpermissions
Snapshots refresh outside the build so no test ever hits the network.

Wire all four checks into the same pipeline stage so a failure blocks the deploy. Report them separately though — a stale snapshot and a genuinely disallowed prefix need different people, and a single opaque “rules check failed” wastes the first ten minutes of every investigation.

Compliance & Operational Guardrails #

  • The parser suite runs on every commit; the configuration suite runs on every commit and on snapshot refresh.
  • Snapshots are refreshed by a scheduled job that fetches politely, honouring the same rate limits as the crawl.
  • A stale snapshot fails the build rather than skipping the assertion.
  • A permission change from allowed to denied pauses the affected crawl scope before the next run.
  • Snapshot text and hash are retained alongside the crawl audit trail as the evidence of what was in force.

Common Mistakes #

  1. Fetching rules files during the test run. Slow, flaky, and it points unexpected traffic at targets from build infrastructure.
  2. Testing only the parser and not the configuration. A perfect parser does not help if the seeds point at disallowed paths.
  3. Skipping when a snapshot is missing. A silent skip is indistinguishable from a pass, and the host with no snapshot is precisely the one nobody has checked.

Frequently Asked Questions #

Should the crawl fail closed if the rules check has not run? #

Yes. Treat a missing or failed check the same way the runtime treats a missing rules file: the host is not crawlable until the position is known. The cost is a delayed run; the alternative is crawling on an assumption nobody verified.

How many fixture cases are enough? #

Start with the dozen above, then add a case every time a real file surprises you. The corpus should grow from production observation rather than from imagination — the strange files sites actually publish are more varied than anything you would invent, and each one added is a behaviour permanently pinned.

Does this replace the runtime rules check? #

No. The runtime check enforces the rules for every request as it happens; the suite verifies that the enforcement behaves as intended and that the configuration is consistent with it. They fail in different ways and both are needed — the pattern is the same one the crawl-delay and sitemap handling guide applies to pacing.

How should the suite handle a host that publishes no rules file at all? #

Snapshot the absence explicitly, with the status and the fetch time, and assert against that rather than skipping. “No rules published” is a real, defensible state — it means nothing is disallowed — and recording it distinguishes a host that has genuinely published nothing from one whose snapshot simply failed to refresh. A skip conflates the two, and the second is the case that matters.

Should the suite fail on a warning rather than an error? #

Split them deliberately. A parser assertion failing is an error and should block the build, because it means the enforcement code no longer behaves as designed. A configured prefix becoming disallowed is a content change rather than a code regression, and blocking every deploy on it means an unrelated change cannot ship while somebody negotiates with a site. Report the second as a failure of the scheduled snapshot job, which pauses that crawl scope and opens a ticket, rather than as a failure of the commit pipeline.