Distributed Crawl Scheduling and Queues: URL Frontiers, Shared Work Queues, and Distributed Politeness #

Distributed crawl scheduling is the discipline of coordinating many worker processes around a single shared URL frontier so that a fleet of crawlers behaves, from the target server’s perspective, like one well-mannered client rather than an uncoordinated swarm. It sits inside the Network Resilience & Proxy Management section because the moment you scale past a single process, correctness stops being about one request and becomes about coordination: which URL runs next, which worker owns it, how fast any single host is being hit across the whole cluster, and how you avoid crawling the same page a thousand times. This guide is for engineers building horizontally scaled crawlers on Redis/RQ, Celery, or SQS who need throughput without trampling the sites they depend on.

Core Principles & Scheduling Foundation #

A distributed crawler is a producer/consumer system wrapped around a URL frontier — the ordered set of URLs discovered but not yet fetched. Workers pull from the frontier, fetch, parse, discover new links, and push those links back. The frontier is where politeness, priority, and deduplication are all enforced, so its design determines whether your crawl is compliant or abusive.

The frontier is not a single queue. A robust design separates two concerns:

• Priority ordering — which URLs should run next (depth, freshness, business value).
• Politeness gating — which URLs are allowed to run next without exceeding a per-host budget.

Classic frontier designs (the Mercator model) split these into a front-end of priority queues and a back-end of per-host queues. A URL is admitted to a host queue only when that host’s rate budget permits, and a heap keyed on “next-allowed time” decides which host is due. You do not need to build Mercator from scratch, but you do need both axes.

Choosing a queue backend #

All three common backends work; they trade operational simplicity against ordering guarantees.

Redis is the most common choice because the same instance holds the frontier, the seen-set, and the per-host rate counters, keeping every scheduling decision on one round-trip. SQS shines when you want the queue managed and durable, but you must implement dedup and politeness in a side store because SQS gives you delivery, not scheduling intelligence.

Implementation Steps #

1. Model the frontier as a priority sorted set #

Use a Redis sorted set where the score encodes both priority and the earliest-eligible timestamp. Lower score runs first.

import time
import redis

r = redis.Redis(decode_responses=True)

def score(priority: int, not_before: float) -> float:
    # Priority dominates; not_before breaks ties and enforces scheduled delays.
    # priority 0 = highest. Scale keeps sub-second ordering within a priority band.
    return priority * 1e9 + not_before

def enqueue(url: str, priority: int = 5, delay: float = 0.0) -> None:
    r.zadd("frontier", {url: score(priority, time.time() + delay)}, nx=True)
    # nx=True: never lower an existing URL's score — first discovery wins,
    # which keeps the crawl deterministic and avoids re-prioritising loops.

2. Deduplicate before you enqueue #

Every discovered link is checked against a shared seen-set before it ever reaches the frontier. This is the single most important compliance control at scale — it caps total requests to a site and prevents crawl loops. The mechanics of canonicalisation, SADD/SISMEMBER, and memory tradeoffs are covered in the companion detail page on deduplicating URLs with a Redis seen-set.

def discover(url: str, priority: int = 5) -> None:
    canonical = canonicalise(url)               # normalise before dedup
    if r.sadd("seen", canonical):               # 1 = newly added, 0 = already seen
        enqueue(canonical, priority=priority)
    # If SADD returns 0 the URL is already known — drop it silently.

3. Gate every fetch through a distributed per-host budget #

Politeness in a single process is a sleep. Politeness across N workers is a shared, atomic rate budget per host, because ten workers each pausing 1s still hit the origin at 10 requests per second. Enforce a per-host token bucket in Redis with a Lua script so the check-and-decrement is atomic. This is the distributed extension of the single-node techniques in implementing polite rate limiting.

# Atomic per-host token bucket. Returns wait-seconds (0 = go now).
RATE_LUA = """
local key = KEYS[1]
local rate = tonumber(ARGV[1])      -- tokens per second (e.g. 0.5 = 1 req / 2s)
local now  = tonumber(ARGV[2])
local last = tonumber(redis.call('HGET', key, 'ts') or now)
local tokens = tonumber(redis.call('HGET', key, 'tk') or '1')
tokens = math.min(1, tokens + (now - last) * rate)
if tokens >= 1 then
    redis.call('HSET', key, 'ts', now, 'tk', tokens - 1)
    redis.call('EXPIRE', key, 3600)
    return 0
end
redis.call('HSET', key, 'ts', now, 'tk', tokens)
return math.ceil((1 - tokens) / rate)
"""
take_token = r.register_script(RATE_LUA)

def host_ready(host: str, rate: float) -> int:
    return int(take_token(keys=[f"rate:{host}"], args=[rate, time.time()]))

4. Lease URLs with a visibility timeout for idempotent workers #

A worker must lease a URL, not delete it. If the worker crashes mid-fetch, the lease expires and the URL returns to the frontier — at-least-once delivery. Combined with the seen-set and idempotent writes to your sink, at-least-once becomes effectively exactly-once. Pop the top eligible URL atomically and re-enqueue it with a future timestamp as an in-flight lease.

POP_LUA = """
local now = tonumber(ARGV[1])
local items = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', '('..(now + 1e9),
                         'LIMIT', 0, 1)
if #items == 0 then return nil end
local url = items[1]
-- Re-score into the future as a lease; a reaper returns expired leases.
redis.call('ZADD', KEYS[1], now + 1e9 + 300, url)
return url
"""
lease = r.register_script(POP_LUA)

def next_url() -> str | None:
    return lease(keys=["frontier"], args=[time.time()])

5. Apply backpressure so producers cannot outrun consumers #

A crawl fans out: one page yields dozens of links. Without backpressure the frontier grows unbounded until Redis evicts keys or OOMs. Cap the frontier depth and have discover shed or defer low-priority URLs when the queue is saturated.

MAX_FRONTIER = 2_000_000

def discover_with_backpressure(url: str, priority: int = 5) -> bool:
    if r.zcard("frontier") >= MAX_FRONTIER and priority > 3:
        return False                    # shed low-priority links under pressure
    discover(url, priority)
    return True

Error Handling & Observability #

Classify failures so the scheduler knows whether to re-lease, defer, or drop a URL. Transient failures return to the frontier with a delay; terminal failures are recorded and dropped.

Emit one structured event per lease outcome:

{
  "event": "fetch_result",
  "url_hash": "9f2c…",
  "host": "example.com",
  "status": 503,
  "attempt": 2,
  "class": "transient",
  "host_qps_1m": 0.48,
  "frontier_depth": 812004,
  "worker_id": "crawler-7",
  "ts": "2026-07-05T09:14:22Z"
}

Track these Prometheus series and alert accordingly:

• crawl_frontier_depth — alert when it grows monotonically for 15m (backpressure failing).
• crawl_host_qps{host} — alert when any host exceeds its configured budget (politeness breach).
• crawl_lease_expired_total — alert on spikes (workers crashing mid-fetch).
• crawl_dedup_hit_ratio — a collapsing ratio signals a canonicalisation bug re-admitting URLs.

Compliance Boundaries #

Distributed politeness is the compliance heart of this pattern. The relevant law and norms — CFAA “exceeds authorized access” exposure in the US, the reasonableness expectations behind GDPR-adjacent enforcement, and site Terms of Service — all turn on whether your aggregate load is reasonable. A single per-worker delay is not a defence when twenty workers sum to a denial-of-service-shaped traffic pattern. Budget the cluster, not the process.

Compute the per-host budget from the site’s declared crawl-delay and divide it across the fleet, not per worker:

cluster_qps_for_host = min(1 / crawl_delay_seconds, configured_ceiling)

Every worker draws from that single shared budget via the token bucket above. Rotating egress IPs does not raise the budget; the budget belongs to the target host, regardless of which proxy the request leaves through. If you rotate egress, do so through ethical proxy rotation systems that preserve, not circumvent, the per-host ceiling.

• Per-host QPS budget is enforced in a shared store, not per worker.
• crawl-delay from robots.txt `crawl-delay` from `robots.txt` sets the ceiling for its host.
• Frontier depth is capped; backpressure sheds load rather than the target.
• 429/Retry-After `429`/`Retry-After` widens the budget cluster-wide, not just for one worker.
• 403/451 `403`/`451`/challenge responses stop crawling that host pending human review.
• Proxy rotation never raises a per-host budget.

Common Mistakes #

1. Per-worker delays instead of a shared budget. Wrong: each worker sleeps 2s. Right: all workers draw tokens from one per-host bucket, so aggregate QPS is capped regardless of worker count.
2. Deleting URLs on pop instead of leasing them. Wrong: ZPOPMIN then crash loses the URL. Right: lease with a visibility timeout and let a reaper return expired leases for at-least-once delivery.
3. Deduplicating after enqueue. Wrong: workers pop and then check the seen-set, wasting frontier space and risking loops. Right: dedup before enqueue so the frontier only ever holds unseen URLs.
4. No frontier cap. Wrong: discover pushes every link unconditionally until Redis OOMs. Right: cap depth and shed low-priority links under backpressure.
5. Treating proxy rotation as a politeness multiplier. Wrong: more IPs, more requests per host. Right: the per-host budget is fixed by the target; proxies change egress, not entitlement.

Frequently Asked Questions #

How do I keep per-domain politeness when workers scale horizontally? #

Move the rate decision out of the worker and into a shared store. A Redis token bucket keyed on the target host, decremented atomically with a Lua script, gives every worker a single source of truth for “may I hit this host right now?”. The per-host QPS ceiling stays constant no matter how many workers you add — adding workers increases parallelism across hosts, never pressure on one host.

Should I use Celery, RQ, or SQS for the frontier? #

Use RQ or a custom Redis sorted set when you want the frontier, seen-set, and rate budget co-located for single-round-trip scheduling. Use Celery when you value its mature retry, routing, and workflow primitives and can keep dedup in a side Redis store. Use SQS when you want a fully managed, durable queue and are willing to implement dedup and politeness in DynamoDB or Redis, since SQS delivers messages but makes no scheduling or dedup decisions for you.

How do I prevent the same URL from being fetched twice? #

Combine three controls: a shared seen-set checked before enqueue, lease-based delivery with visibility timeouts so a crash re-queues rather than duplicates, and idempotent writes to your sink keyed on canonical URL or content hash. At-least-once delivery plus idempotent writes yields effectively exactly-once processing without a distributed transaction.

What is backpressure and why does a crawler need it? #

Backpressure is the mechanism that stops producers from overwhelming consumers. A crawl amplifies — one fetched page discovers many links — so an unbounded frontier grows until your queue store runs out of memory. Capping frontier depth and shedding or deferring low-priority discoveries when the cap is hit keeps the system stable and, as a side effect, keeps you from queuing far more load than the target could ever tolerate.

Related guides #

• Network Resilience & Proxy Management — the parent section on scaling crawlers without breaking them or their targets.
• Deduplicating URLs with a Redis seen-set — the canonicalisation and seen-set mechanics this scheduler depends on.
• Implementing polite rate limiting — the single-node rate-limiting foundation that the distributed token bucket extends.
• Building ethical proxy rotation systems — how to rotate egress without inflating per-host budgets.
• Exponential backoff and retry logic — the delay strategy for re-enqueuing transient failures.