Compliant Web Scraping

Handling Crawl-delay and Sitemap: Directives in robots.txt #

The two most useful lines in many robots.txt files — Crawl-delay and Sitemap: — are the two that Python’s standard urllib.robotparser handles least. This page shows how to extract both reliably by reading raw directive lines, apply Crawl-delay to your request pacing, and seed a crawl frontier from the sitemaps a site advertises. It sits under parsing robots.txt programmatically within the Compliance & Ethical Crawling Foundations section, and it picks up where basic allow/deny parsing leaves off: turning the non-Allow/Disallow directives into concrete pacing and scheduling behaviour.

Problem Framing #

Crawl-delay and Sitemap: are extensions, not part of the original robots exclusion standard, and tooling treats them inconsistently. urllib.robotparser added a crawl_delay() accessor in Python 3.6 and a site_maps() accessor, but both are brittle: crawl_delay() returns None for the wildcard * group in several real files it fails to associate correctly, and site_maps() returns None rather than an empty list when no sitemap is present, which trips up naive callers. Neither gives you the raw line if parsing quietly drops it.

In production this surfaces as two silent failures. First, a site politely asks for a ten-second gap between requests and your crawler ignores it because the parser handed back None, so you hammer the endpoint and earn a block. Second, a site publishes a comprehensive sitemap index that would let you discover every URL cheaply, and you never read it because your parser only exposed Disallow rules. Both are avoidable by reading the file’s raw lines alongside the standard parser.

Step-by-Step Implementation #

  1. Fetch robots.txt once, keep the raw text. You need the bytes both for the standard parser and for line-level extraction of the non-standard directives.
  2. Feed the text to urllib.robotparser for authoritative can_fetch() allow/deny decisions — do not reimplement that logic.
  3. Scan raw lines yourself for Crawl-delay and Sitemap, because these are the fields the standard parser handles unreliably. Match case-insensitively and split on the first colon.
  4. Resolve the applicable Crawl-delay by tracking which User-agent group each directive belongs to, preferring a group matching your agent token over the * fallback.
  5. Collect every Sitemap: URL — these are global to the file, not scoped to a user-agent group — and hand them to your frontier seeder.
import urllib.robotparser
import urllib.request

def parse_robots(robots_url: str, my_agent: str):
    raw = urllib.request.urlopen(robots_url, timeout=10).read().decode("utf-8", "replace")

    # Standard parser for authoritative allow/deny decisions.
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(raw.splitlines())

    sitemaps, crawl_delays, current_agents = [], {}, ["*"]
    for line in raw.splitlines():
        line = line.split("#", 1)[0].strip()      # strip comments
        if not line or ":" not in line:
            continue
        field, _, value = line.partition(":")
        field, value = field.strip().lower(), value.strip()
        if field == "user-agent":
            current_agents = [value.lower()]
        elif field == "crawl-delay":
            for agent in current_agents:
                crawl_delays[agent] = float(value)   # seconds between requests
        elif field == "sitemap":
            sitemaps.append(value)                   # absolute URL, file-global

    # Prefer a delay targeting our token; fall back to the wildcard group.
    delay = crawl_delays.get(my_agent.lower(), crawl_delays.get("*"))
    return rp, delay, sitemaps

Applying Crawl-delay to pacing #

A Crawl-delay of N means “leave at least N seconds between the start of consecutive requests to this host.” Feed it into your limiter as a per-host minimum interval rather than a blind sleep, so it composes cleanly with the rest of your polite rate limiting:

import time

class HostPacer:
    def __init__(self, crawl_delay: float | None, floor: float = 1.0):
        # Honour the site's delay, but never crawl faster than our own floor.
        self.interval = max(crawl_delay or 0.0, floor)
        self._last = 0.0

    def wait(self):
        gap = time.monotonic() - self._last
        if gap < self.interval:
            time.sleep(self.interval - gap)
        self._last = time.monotonic()

Seeding the frontier from sitemaps #

Sitemap URLs may point at a <sitemapindex> (a list of child sitemaps) or a <urlset> (actual page URLs). Resolve the index, then enqueue the leaf URLs into your scheduler — the same frontier you would manage with distributed crawl scheduling and queues:

import gzip, requests
from lxml import etree

def iter_sitemap_urls(sitemap_url: str):
    body = requests.get(sitemap_url, timeout=15).content
    if sitemap_url.endswith(".gz"):
        body = gzip.decompress(body)
    root = etree.fromstring(body)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall(".//sm:sitemap/sm:loc", ns):
            yield from iter_sitemap_urls(loc.text.strip())   # recurse into children
    else:
        for loc in root.findall(".//sm:url/sm:loc", ns):
            yield loc.text.strip()

Verification & Testing #

Point the parser at a fixture and assert both directives resolve. A minimal robots.txt served from a local file or httpbin-style echo makes this deterministic:

FIXTURE = """User-agent: *
Crawl-delay: 10
Disallow: /private

Sitemap: https://example.com/sitemap_index.xml
"""

def test_extracts_delay_and_sitemap(tmp_path):
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(FIXTURE.splitlines())
    # Re-run the raw scan from parse_robots against FIXTURE...
    assert delay == 10.0
    assert "https://example.com/sitemap_index.xml" in sitemaps
    assert rp.can_fetch("*", "https://example.com/public") is True

To confirm pacing behaves, log timestamps across ten HostPacer.wait() calls and assert consecutive gaps are >= interval. To confirm sitemap seeding, count the URLs yielded from a known sitemap and compare against the <loc> count in the raw XML.

Compliance & Operational Guardrails #

  • Crawl-delay is a request, but honour it as a floor. Treating it as advisory and ignoring it is precisely the behaviour that gets crawlers characterised as abusive; enforce the larger of the site’s value and your own polite baseline.
  • A published sitemap is an invitation, not blanket consent. It tells you which URLs exist, but Disallow rules and Terms of Service still govern which you may fetch — always gate sitemap URLs through can_fetch().
  • Cap absurd delays. A malformed or hostile Crawl-delay: 86400 should trip a review rather than silently stalling your pipeline for a day; clamp to a sane ceiling and alert.

Common Mistakes #

  1. Trusting crawl_delay() alone. It returns None for cases where the directive is plainly present in the file; always cross-check with a raw-line scan before assuming a site set no delay.
  2. Scoping Sitemap: to a user-agent group. Sitemap directives are file-global; associating them with the nearest User-agent block drops sitemaps declared before any group.
  3. Enqueuing sitemap URLs without deduplication. Sitemap indexes overlap and repeat; feed the URLs through your seen-set before they hit the frontier or you will re-crawl thousands of duplicates.

Frequently Asked Questions #

Does Google honour Crawl-delay? #

No — Googlebot explicitly ignores Crawl-delay and manages its own rate via Search Console instead. That does not make the directive safe to ignore for your own crawler: many smaller sites rely on it as their only throttling signal, and disregarding it is exactly the conduct that provokes blocks and complaints.

Where should I look for sitemaps if robots.txt has no Sitemap: line? #

Try the conventional /sitemap.xml at the site root, then check for a <link rel="sitemap"> in page markup. Absence of any advertised sitemap is not permission to crawl more aggressively — fall back to polite link discovery within the allowed paths.

How do I handle a gzipped or nested sitemap index? #

Decompress .gz payloads before parsing, and recurse when the root element is <sitemapindex> rather than <urlset>, as shown above. Guard the recursion with a depth limit and your seen-set so a malformed index that references itself cannot loop forever.