Compliant Web Scraping

Deduplicating URLs with a Redis Seen-Set #

The narrow question here is: across a fleet of crawler workers, how do you decide in a single fast operation whether a URL has already been queued or fetched, so you never crawl the same page twice? This detail page belongs to the Distributed Crawl Scheduling and Queues technique within the Network Resilience & Proxy Management section, and it covers the two production answers — an exact Redis SET and a probabilistic RedisBloom filter — along with the URL canonicalisation that must precede both.

Problem Framing #

URL-level deduplication is what caps the total number of requests a distributed crawl sends to a site. Web graphs are dense and cyclic: the same product page is linked from a category listing, a breadcrumb, a “related items” rail, and pagination, so a naive crawler re-discovers it dozens of times. Without a shared record of what has been seen, every worker re-enqueues every rediscovery, the frontier explodes, and the target host absorbs the same fetch repeatedly — wasteful for you and abusive to them.

The seen-set must be shared and atomic. A per-worker Python set is useless when ten workers each discover the same link. And the check must be a check-and-set in one operation, or two workers racing on the same new URL will both conclude “not seen” and both enqueue it. Redis gives you exactly this: SADD returns whether the member was newly added, atomically, cluster-wide.

Step-by-Step Implementation #

  1. Canonicalise before you dedup. https://EXAMPLE.com/p?b=2&a=1 and https://example.com/p?a=1&b=2#reviews are the same resource. Normalise scheme/host case, drop the fragment, sort query parameters, strip tracking params, and remove default ports. Skipping this step is the number-one cause of a “working” dedup that still re-crawls.
  2. Hash the canonical URL to a fixed-width digest so set members are small and uniform regardless of URL length.
  3. SADD the digest; a return of 1 means newly seen (enqueue it), 0 means already seen (drop it).
  4. For very large crawls, switch the backing store to a RedisBloom filter to trade a tiny false-positive rate for a large memory saving.
import hashlib
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
import redis

r = redis.Redis(decode_responses=True)

TRACKING = {"utm_source", "utm_medium", "utm_campaign", "utm_term",
            "utm_content", "gclid", "fbclid", "ref"}

def canonicalise(url: str) -> str:
    parts = urlsplit(url.strip())
    host = parts.hostname or ""
    # Drop default ports; lowercase scheme + host (path stays case-sensitive).
    netloc = host.lower()
    if parts.port and not ((parts.scheme == "http" and parts.port == 80)
                           or (parts.scheme == "https" and parts.port == 443)):
        netloc = f"{netloc}:{parts.port}"
    # Sort params and strip tracking noise so equivalent URLs collapse to one key.
    query = urlencode(sorted(
        (k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
        if k not in TRACKING
    ))
    path = parts.path or "/"
    return urlunsplit((parts.scheme.lower(), netloc, path, query, ""))  # no fragment

def url_digest(url: str) -> str:
    # blake2b at 16 bytes = 128-bit key: collision-safe for billions of URLs.
    return hashlib.blake2b(canonicalise(url).encode(), digest_size=16).hexdigest()

def mark_seen(url: str) -> bool:
    """Return True if this URL is new (enqueue it), False if already seen."""
    return bool(r.sadd("seen:urls", url_digest(url)))

For a Bloom filter, RedisBloom’s BF.ADD returns the same new/seen signal with bounded memory. Provision it once with a target capacity and error rate:

# Reserve a filter for 100M URLs at a 0.1% false-positive rate (~180 MB).
redis-cli BF.RESERVE seen:bloom 0.001 100000000
def mark_seen_bloom(url: str) -> bool:
    # BF.ADD returns 1 if newly added, 0 if (probably) already present.
    return bool(r.execute_command("BF.ADD", "seen:bloom", url_digest(url)))

The memory tradeoff is stark. An exact SET of 128-bit digests costs roughly 60–80 bytes per URL once Redis overhead is included — about 7 GB for 100M URLs. A Bloom filter at a 0.1% false-positive rate holds the same 100M URLs in under 200 MB, a ~35x saving. The price is one-directional false positives: the filter may occasionally report a genuinely new URL as “seen” and skip it, but it will never claim a seen URL is new. For a crawl, skipping an occasional page is usually acceptable; re-fetching one is the failure you were trying to prevent, and Bloom never does that.

Verification & Testing #

Confirm canonicalisation collapses equivalents and that the second insert reports “seen”:

def test_dedup_collapses_equivalents():
    a = "https://EXAMPLE.com:443/p?b=2&a=1#reviews"
    b = "https://example.com/p?a=1&b=2&utm_source=news"
    assert url_digest(a) == url_digest(b)     # same canonical resource

    r.delete("seen:urls")
    assert mark_seen(a) is True               # first sighting -> enqueue
    assert mark_seen(b) is False              # equivalent -> dropped

Check the live seen-set size and estimated false positives from the CLI:

redis-cli SCARD seen:urls          # exact set cardinality
redis-cli BF.INFO seen:bloom       # capacity, size, and items inserted

Watch the dedup hit ratio in Prometheus — crawl_dedup_hit_ratio = seen_hits / discoveries. A ratio that collapses toward zero means canonicalisation regressed and equivalent URLs are slipping through as distinct keys.

Compliance & Operational Guardrails #

  • Dedup is a load-limiting control, not just an efficiency trick. It bounds total requests to a host, which is central to the reasonableness expected under site Terms of Service and to avoiding CFAA-style “excessive access” exposure. Treat a broken seen-set as a compliance incident, not a performance bug.
  • Store digests, never raw URLs with embedded identifiers. URLs can carry session tokens, emails, or personal identifiers in query strings; hashing to a fixed digest keeps the seen-set free of retained personal data and aligns with data-minimisation.
  • Persist the seen-set across restarts (Redis AOF/RDB or a rebuildable checkpoint). Losing it silently re-authorises a full re-crawl of a site you already fetched, doubling your footprint on the target.
  • Bound the filter’s lifetime. A permanent seen-set means you never re-visit for freshness; scope it per crawl or expire it on a cadence that matches your recrawl policy.

Common Mistakes #

  1. Deduplicating raw URLs without canonicalisation, so ?a=1&b=2 and ?b=2&a=1 register as two pages and the crawl re-fetches the same resource.
  2. Using a per-process set that each worker keeps privately, defeating deduplication the moment you scale past one worker.
  3. Choosing a Bloom filter capacity far below the real URL count, which drives the false-positive rate up until the filter starts skipping large numbers of genuinely new pages.

Frequently Asked Questions #

When should I use a Bloom filter instead of a plain Redis SET? #

Switch to RedisBloom when the exact set no longer fits comfortably in memory — roughly tens of millions of URLs and up, where a SET runs into multiple gigabytes. If you can tolerate skipping a small fraction of new pages (the false-positive cost) in exchange for a ~35x memory reduction, the Bloom filter wins. Below a few million URLs, the exact SET is simpler and its “never skip a new page” guarantee is worth the RAM.

Does URL-level dedup replace record-level deduplication of the scraped data? #

No — they solve different problems. URL dedup stops you fetching the same page twice; it says nothing about whether two different URLs yielded the same content or whether a record already exists in your warehouse. For deduplicating the extracted data itself, see the record-level deduplication strategies for scraped data, which uses content hashing and key matching downstream of the crawl.

Can false positives cause me to miss important pages? #

Yes, but the effect is bounded and tunable. At a 0.1% false-positive rate, roughly one in a thousand new URLs may be wrongly skipped. Lower the configured error rate (at the cost of more memory) for crawls where completeness matters, or run an exact SET for high-value hosts and a Bloom filter for the vast, low-priority remainder.