Headless Browser Resource Management #

A headless browser is the most expensive component a crawl can contain: hundreds of megabytes of memory per instance, a request footprint several times larger than a plain fetch, and a startup cost measured in hundreds of milliseconds. Used deliberately it unlocks content no HTTP client can reach; used by default it multiplies both your infrastructure bill and the load you place on every target. This topic, part of the Network Resilience & Proxy Management section, covers when a browser is genuinely required, how to bound what each instance consumes, and how to run a pool of them without the memory growth that eventually kills every naive implementation.

Core Principles: Deciding Whether to Render at All #

The first resource decision is whether to spend a browser on a page. Three checks, in order, resolve most cases without one.

Each check that passes avoids a browser process and most of the requests.

Is the data already in the HTML? Client-rendered pages very often ship their initial state as a serialised payload in a script element. If it is there, the extraction is a JSON normalisation problem and the browser is unnecessary.

Is there a data endpoint? The request the page’s own JavaScript makes to populate itself is usually a plain, documented-enough JSON API. Calling it directly is cheaper for both sides and dramatically more stable than parsing rendered markup.

Is there a non-rendered variant? Print views, feed endpoints, and pagination fallbacks frequently expose the same content without scripts.

Only when all three fail is a browser the right tool. Encoding this as a routing decision rather than a per-spider habit keeps the expensive path rare.

from dataclasses import dataclass

@dataclass(frozen=True)
class RenderDecision:
    render: bool
    reason: str

def decide_render(html: str, site_config) -> RenderDecision:
    if site_config.force_render:
        return RenderDecision(True, "site configured to always render")
    if embedded_payloads(html):
        return RenderDecision(False, "initial state found in the document")
    if site_config.data_endpoint_template:
        return RenderDecision(False, "data endpoint configured")
    if content_selector_matches(html, site_config.content_selector):
        return RenderDecision(False, "content present in the served markup")
    return RenderDecision(True, "content absent without script execution")

Log the reason on every decision and export the render rate per host. A host whose render rate climbs from 5% to 90% has changed its delivery, and the cost of that change is worth noticing on the day it happens rather than in the following month’s invoice.

The Three Levels of Isolation #

Browser automation offers three nested scopes, and using the wrong one is the root cause of most memory problems.

A browser process is expensive to start and holds the shared network stack, cache and GPU resources. Start few, keep them alive, and reuse them.

A browser context is cheap, and it is the real isolation boundary: cookies, storage and permissions are per-context. One context per crawl unit — a page, a session, a site — gives clean isolation for a fraction of the cost of a new process.

A page is cheapest of all, and it shares its context’s state with sibling pages. Pages are for tabs within one logical session, not for isolating unrelated work.

The standard mistake is one browser process per page, which multiplies startup cost and memory by the page count. The second mistake is one context for everything, which leaks cookies and session state between unrelated targets. One long-lived process, one context per unit of work, closed promptly, is the shape that works.

Bounding What Each Instance Consumes #

Cut the request footprint #

One long-lived process, one context per unit of work, closed promptly.

A rendered page fetches everything a real browser would: images, fonts, stylesheets, analytics, advertising. For a crawler that wants text, most of that is pure waste — waste that the target pays for in bandwidth as well.

BLOCKED_TYPES = {"image", "media", "font"}
BLOCKED_HOSTS = ("googletagmanager.com", "google-analytics.com", "doubleclick.net",
                 "facebook.net", "hotjar.com", "segment.io")

async def install_request_filter(context) -> None:
    async def route(handler):
        request = handler.request
        if request.resource_type in BLOCKED_TYPES:
            await handler.abort()
            return
        if any(host in request.url for host in BLOCKED_HOSTS):
            await handler.abort()
            return
        await handler.continue_()
    await context.route("**/*", route)

Blocking images, fonts and third-party trackers typically removes 60–80% of a page’s bytes and a similar share of its request count, and it usually makes the page render faster because fewer resources compete for the network. Do not block stylesheets indiscriminately: layout-dependent extraction and screenshot capture both need them, and a page rendered without CSS can collapse to a structure your selectors do not match. The measurements and the exceptions are worked through in blocking images and fonts to cut crawl bandwidth.

Bound the wait #

The single largest source of wasted browser time is waiting for a network-idle condition on a page that never goes idle — a polling widget, a live ticker, an advertising auction. Wait for the selector you need, with a deadline, rather than for a global condition.

async def render_and_extract(context, url: str, ready_selector: str, timeout_ms: int = 15000):
    page = await context.new_page()
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=timeout_ms)
        await page.wait_for_selector(ready_selector, timeout=timeout_ms, state="attached")
        return await page.content()
    finally:
        await page.close()

domcontentloaded plus an explicit selector is both faster and more deterministic than networkidle, and it fails in a way you can act on: a timeout waiting for a known selector means the page changed, which is a useful signal, whereas a networkidle timeout means almost nothing.

Recycle before it matters #

Browser processes accumulate memory over a long run — caches, detached nodes, extension state. Rather than diagnosing each leak, recycle on a bounded lifetime.

from dataclasses import dataclass, field
import time

@dataclass
class BrowserLease:
    browser: object
    started_at: float = field(default_factory=time.monotonic)
    pages_served: int = 0
    max_pages: int = 500
    max_age_seconds: float = 1800.0

    def is_exhausted(self) -> bool:
        return (self.pages_served >= self.max_pages
                or time.monotonic() - self.started_at > self.max_age_seconds)

Recycling on both a page count and an age bounds the two ways a process degrades. Retire the browser between crawl units rather than mid-page, and keep a spare warm so the swap does not stall the pool — the pooling mechanics are covered in limiting Playwright memory with browser contexts.

Compliance Boundaries #

A browser can do things a plain client cannot, and some of them are the wrong side of the line drawn in the rest of this section.

Every allowed subrequest is charged to the host budget, not just the navigation.

Rendering is used to read content, never to solve or bypass a challenge
The crawler token identifies the browser stage exactly as it does the HTTP stage
Blocked resource types are chosen to reduce load, not to disguise the client
Session state belongs to your own authorised account, never to another person
Rendered pages are subject to the same rules-file and rate checks as plain fetches
Script execution does not extend to interacting with forms that create data on the target

The last item deserves emphasis. A browser makes it trivial to submit a form, and a crawl that begins submitting searches or filters is no longer reading published content — it is generating work on the target’s infrastructure, at a cost per request far above a page view. Where a search interface genuinely is the only route to the data, that is a conversation to have with the operator rather than a capability to exercise quietly.

Rate limiting applies identically to rendered pages, and rendering makes it more important rather than less: one rendered page can produce dozens of requests, so a limiter counting page loads rather than requests will undercount your actual footprint by an order of magnitude. Count requests, not navigations, when reporting against a host budget set by the polite rate limiting layer.

Sizing a Browser Fleet #

Browser capacity is the most expensive part of a crawl to over-provision and the most disruptive to under-provision, and the arithmetic is straightforward once the right numbers are measured.

Start from throughput. A rendered page costs roughly the sum of navigation time, the wait for the ready selector, and the extraction — typically two to six seconds with a request filter in place, against two hundred milliseconds for a plain fetch. At four seconds per page and four concurrent contexts, one worker renders about 3,600 pages an hour.

Then bound by memory, which is the real constraint. Measure resident memory per browser process under your actual workload rather than trusting a figure from documentation: a text-extraction crawl with images blocked sits far lower than one rendering media-heavy pages.

def fleet_plan(pages_per_hour: int, seconds_per_page: float,
               mb_per_browser: int, contexts_per_browser: int,
               worker_memory_mb: int, headroom: float = 0.3) -> dict:
    """Workers and browsers needed, bounded by memory rather than by ambition."""
    concurrent_needed = (pages_per_hour * seconds_per_page) / 3600
    browsers_needed = max(1, round(concurrent_needed / contexts_per_browser + 0.5))

    usable_mb = worker_memory_mb * (1 - headroom)
    browsers_per_worker = max(1, int(usable_mb // mb_per_browser))
    workers = max(1, -(-browsers_needed // browsers_per_worker))    # ceiling division

    return {
        "concurrent_contexts": round(concurrent_needed, 1),
        "browsers": browsers_needed,
        "browsers_per_worker": browsers_per_worker,
        "workers": workers,
    }

The 30% headroom is not conservatism for its own sake. A browser pool that reaches its memory ceiling does not degrade gracefully — the process is killed, taking every in-flight context with it — so the difference between 70% and 95% utilisation is the difference between a slightly larger bill and a crawl that loses work at unpredictable intervals.

Two further multipliers are easy to forget. Screenshot capture roughly doubles peak memory during the capture itself, so a fleet that occasionally screenshots needs headroom sized for the screenshot rather than the average. And pages that legitimately need images unblocked — lazy-loading layouts — consume substantially more raster memory, which argues for routing them to a separate, smaller pool rather than raising the ceiling everywhere.

Failure Modes Under Load #

Three symptoms indicate a fleet that is sized wrong, and each points somewhere different.

Steadily rising memory on a stable process count means process ageing: the recycle limits are too generous. Lower the maximum context count or the maximum age.

Rising process count means leases are failing to be retired, or crashed browsers are not being replaced. Check that retirement only skips leases with active contexts, and that a launch failure raises rather than silently reducing capacity.

Rising queue wait with flat memory and processor use means the pool is too small for the offered load, and the answer is more workers rather than more contexts per browser — contexts share a process, and a busy process becomes the bottleneck well before its memory ceiling.

Export browsers, contexts, resident memory and context-acquisition wait as gauges, and the diagnosis takes seconds rather than an afternoon.

Deciding Per Page Type, Not Per Site #

The render decision belongs at the page-type level, because most sites are a mixture. A marketplace commonly serves server-rendered category pages and client-rendered detail pages; an editorial site serves static articles and dynamic live blogs. Rendering the whole site because part of it needs it multiplies cost by the ratio of static to dynamic pages, which is frequently ten to one.

# selectors/marketplace.example.com.yaml (excerpt)
page_types:
  category:
    match: "//meta[@property='og:type'][@content='website']"
    render: never                      # server-rendered; a plain fetch suffices
  product:
    match: "//meta[@property='og:type'][@content='product']"
    render: if_missing                 # try plain, render only when the selector is absent
    content_selector: "[data-testid='product-price']"
  live:
    match: "//*[@data-live-region]"
    render: always
    ready_selector: "[data-testid='feed-item']"

if_missing is the setting that earns the most. Fetch plainly, check whether the content selector matched, and render only when it did not — which handles sites that are progressively enhanced, where most pages arrive complete and a minority do not. Record which path each fetch took, and the render rate per page type becomes a metric: a rate climbing from 5% to 90% means the site has changed its delivery, and the cost of that change is visible on the day rather than in the next invoice.

Common Mistakes #

One browser process per page. Startup cost and memory scale with the page count, and the pool spends most of its time launching processes.
Never closing contexts. Memory grows monotonically until the worker is killed, and the leak is invisible in per-page measurements.
Waiting for network idle. Pages with polling never go idle, so every fetch pays the full timeout.
Blocking stylesheets by default. Layout-dependent selectors stop matching and screenshots become useless.
Rendering every page because some pages need it. The render decision belongs per page type, driven by whether the content is actually absent.

Frequently Asked Questions #

How many concurrent browsers can one worker run? #

Fewer than most teams expect. A modest instance with 4 GB of memory supports roughly two to four concurrent browser processes with several contexts each, and the limit is memory rather than processor. Measure resident memory per instance under your real workload and set the pool from that, leaving headroom — a browser pool that hits the memory ceiling does not degrade, it gets killed.

Is a headless browser detectable, and does that matter? #

It is detectable, and it should not matter, because a compliant crawler is not trying to hide. Send your crawler token, publish your contact route, and respect the site’s rules; if a site chooses to refuse rendered clients, that refusal is an answer to be honoured rather than a detection problem to be solved.

Should screenshots be captured as part of a crawl? #

Only where the rendered output is itself the subject — visual regression checks, layout research, evidence of what a page displayed. Screenshots are large, they frequently contain personal data that the extracted record does not, and they inherit the same retention obligations as any other stored artefact. Capture deliberately, store briefly, and never as a default “just in case”.

Does rendering change what the rules file permits? #

No. The rules apply to the URL, not to the client, and a page that may not be fetched by an HTTP client may not be rendered either. In practice a browser makes the check more important, because rendering pulls in subresources whose URLs your crawler never explicitly chose — filter those against the same rules before allowing them.

How should a browser stage be tested? #

Against saved pages served from a local static server, not against live sites. A rendered fixture — the HTML plus the scripts and styles it needs — reproduces the rendering path deterministically, runs in milliseconds, and does not send test traffic to a third party every time somebody pushes a commit. Reserve live runs for a small scheduled smoke test that confirms the pages you actually crawl still behave as the fixtures assume.

Is it worth running the browser stage on separate infrastructure? #

Usually yes, once it is more than a small fraction of the crawl. Browser workers have a completely different resource profile from HTTP workers — memory-bound rather than network-bound — and mixing them means sizing one machine for both and over-provisioning for each. Separating them also contains the failure: a browser pool exhausting its memory takes down browser work only, while the plain HTTP crawl continues.

Yes, and it should share the rate limiter too. From a target’s perspective there is one crawler, and splitting the accounting between two stages is how a host ends up receiving twice the agreed rate while both stages report compliance. One limiter, one budget, one identity — the stages differ only in how they fetch, not in how much they are permitted to.

Limiting Playwright Memory With Browser Contexts — pooling and recycling without unbounded growth.
Blocking Images and Fonts to Cut Crawl Bandwidth — the request filter and what it actually saves.
Extracting Tables From Dynamic JavaScript Pages — checking for an embedded payload before rendering at all.
Session Cookie Management in Headless Browsers — keeping context state contained and verifiable.