Blocking Images and Fonts to Cut Crawl Bandwidth #

A rendered page fetches everything a person’s browser would, and a crawler extracting text needs almost none of it. Filtering resource requests is the single highest-return optimisation available to a browser-based crawl: it typically removes most of the bytes and a large share of the requests, and the load it removes is load the target was paying for. This guide implements the filter and shows where it must not be applied, as part of headless browser resource management in the Network Resilience & Proxy Management section.

Problem Framing #

A typical content page issues 60 to 150 subrequests. Images account for most of the bytes, fonts for a surprising share of the latency because they block text rendering, and third-party analytics and advertising for a trailing spread of requests to hosts that have nothing to do with the content.

This matters on both sides of the connection. For the crawler it is bandwidth, memory for decoded images, and wall-clock time. For the target it is real serving cost — and, importantly, it is cost your rate limiter is probably not counting. A limiter configured for ten page loads per minute against a page issuing a hundred subrequests is permitting a thousand requests per minute, which is not what anyone intended when the budget was set.

Filtering therefore serves politeness as directly as it serves throughput.

Step-by-Step Implementation #

1. Filter by resource type first #

BLOCKED_TYPES = {"image", "media", "font"}

async def install_request_filter(context) -> None:
    async def handler(route):
        request = route.request
        if request.resource_type in BLOCKED_TYPES:
            await route.abort()
            return
        await route.continue_()
    await context.route("**/*", handler)

Resource type is the browser’s own classification and is far more reliable than matching file extensions, which fail on the very common case of an image served from a path with no extension.

Note what is not blocked: document, stylesheet, script, xhr, and fetch. Scripts and data requests are what populate the content you came for, and stylesheets matter more than they appear to — see below.

2. Add a third-party host filter #

Analytics, tag managers, advertising exchanges and session-recording tools contribute requests and latency and nothing else. Blocking them by host is both effective and easy to keep current.

BLOCKED_HOSTS = (
    "googletagmanager.com", "google-analytics.com", "analytics.google.com",
    "doubleclick.net", "googlesyndication.com", "adservice.google.",
    "facebook.net", "connect.facebook.net",
    "hotjar.com", "fullstory.com", "segment.io", "segment.com",
    "optimizely.com", "mouseflow.com", "clarity.ms",
)

def is_blocked_host(url: str) -> bool:
    return any(host in url for host in BLOCKED_HOSTS)

Blocking these is not evasion — quite the opposite. A crawler is not a visitor, and allowing analytics to fire pollutes the target’s own numbers with traffic that is not a person. Filtering them is more considerate than letting them through.

3. Keep stylesheets unless you have measured that you can drop them #

This is the exception people get wrong. Blocking CSS looks attractive — stylesheets are large and text does not need styling — but three things break.

Layout-dependent extraction. Selectors written against a rendered layout, and any logic that reads element positions or visibility, behave differently without CSS.

Visibility checks. Content hidden by CSS is a signal your extractor may rely on, including the soft paywall detection that distinguishes a preview from a full document.

Screenshots. An unstyled screenshot is worthless as evidence of what the page displayed.

If a site’s extraction is purely structural and you never screenshot it, blocking stylesheets is safe and worth doing — but make it a per-site setting proven against fixtures, not a global default.

from dataclasses import dataclass

@dataclass(frozen=True)
class FilterPolicy:
    block_images: bool = True
    block_fonts: bool = True
    block_media: bool = True
    block_stylesheets: bool = False        # per-site, only where proven safe
    block_third_party: bool = True
    allow_hosts: tuple[str, ...] = ()      # e.g. a CDN that serves the data endpoint

def blocked_types(policy: FilterPolicy) -> set[str]:
    types = set()
    if policy.block_images: types.add("image")
    if policy.block_fonts: types.add("font")
    if policy.block_media: types.add("media")
    if policy.block_stylesheets: types.add("stylesheet")
    return types

4. Count what you allowed, and charge it to the host budget #

from collections import Counter

async def install_counting_filter(context, policy: FilterPolicy, budget) -> Counter:
    stats = Counter()
    types = blocked_types(policy)

    async def handler(route):
        request = route.request
        if request.resource_type in types:
            stats["blocked_type"] += 1
            await route.abort(); return
        if policy.block_third_party and is_blocked_host(request.url) \
                and not any(a in request.url for a in policy.allow_hosts):
            stats["blocked_host"] += 1
            await route.abort(); return

        stats["allowed"] += 1
        budget.charge(host_of(request.url))     # every allowed request counts
        await route.continue_()

    await context.route("**/*", handler)
    return stats

Charging every allowed subrequest to the host budget is what makes a rendered crawl honest about its footprint. Without it, the limiter counts one request per page and the target sees forty.

Verification & Testing #

Measure on real pages from your own target set — the savings vary enormously between a text-heavy article and an image gallery.

@pytest.mark.asyncio
async def test_filter_reduces_requests_and_bytes(pool, sample_url):
    async with pool.context() as bare:
        unfiltered = await measure(bare, sample_url)
    async with pool.context() as filtered:
        await install_request_filter(filtered)
        filtered_result = await measure(filtered, sample_url)

    assert filtered_result.requests < unfiltered.requests * 0.5
    assert filtered_result.bytes < unfiltered.bytes * 0.4
    # Most important: the extraction still works.
    assert filtered_result.record == unfiltered.record

The final assertion is the one that keeps the optimisation honest. A filter that halves the bytes and changes the extracted record has not saved anything; it has introduced a silent data bug. Run it across a fixture per page type before enabling a new policy for a site.

Typical results for a text-focused crawl: requests down by 55–75%, bytes down by 70–85%, and page render time down by a third. Sites that lazy-load content on scroll are the exception — blocking images can prevent an intersection observer from firing, and the content never loads. Where that happens, allow images for that site and record why.

Compliance & Operational Guardrails #

• Every allowed subrequest is charged to the host’s rate budget, not just the navigation.
• Filtering is used to reduce load, never to alter how the client presents itself.
• Analytics and tracker requests are blocked so the target’s own metrics are not polluted by a bot.
• The filter policy is per site, versioned in configuration, and proven against fixtures before rollout.
• Aborted request counts are exported, so a policy that is silently breaking a site is visible.

Common Mistakes #

1. Blocking stylesheets globally. Layout-dependent selectors and visibility checks stop working, and screenshots become useless.
2. Blocking by file extension. Modern asset URLs frequently carry no extension, so the filter misses most of what it targets.
3. Not counting allowed subrequests against the budget. The crawler’s real footprint is an order of magnitude larger than its metrics claim.

Frequently Asked Questions #

Does blocking resources make the crawler more detectable? #

Possibly, and it does not matter. A compliant crawler identifies itself in its token and publishes a contact route, so there is nothing to detect that you have not already declared. Blocking resources is a bandwidth decision, and reducing the load you place on a target is not something to conceal.

What about pages that lazy-load content when images appear? #

Allow images for those sites. An intersection observer watching image elements will not fire if the images never load, and the content you came for never appears. This is exactly why the policy is per site and why the fixture comparison asserts on the extracted record rather than only on the byte savings.

Should the same filtering apply to the plain HTTP stage? #

There is nothing to filter — an HTTP client fetches only the document you asked for. That asymmetry is itself a good reason to prefer the plain path wherever the content allows it, as the render decision covers.

Should the filter block requests to the site’s own subdomains? #

No, unless a specific subdomain is demonstrably serving only tracking. Content, data endpoints and required scripts frequently live on a separate asset or API subdomain, and blocking by domain rather than by resource type breaks pages in ways that are hard to attribute. Type-based blocking is precise; host-based blocking should be reserved for the third-party trackers where the host is genuinely the identifying feature.

How should the policy be validated after a site changes? #

Re-run the paired comparison — filtered against unfiltered — on a fresh sample and assert that the extracted records still match. Doing this on the same schedule as the fixture refresh catches the case where a site moves its content behind a resource type the filter blocks. The failure mode without that check is subtle: bandwidth savings look better than ever, because the page is now loading almost nothing, and the extraction quietly returns less.

Related guides #

• Headless Browser Resource Management — deciding when a browser is needed at all.
• Limiting Playwright Memory With Browser Contexts — the pool that installs this filter on every context.
• Implementing Polite Rate Limiting — the budget that allowed subrequests must be charged against.