Limiting Playwright Memory With Browser Contexts #

A browser pool that grows its memory footprint all day has one of three bugs, and all three are cheap to fix once you know which one you have: contexts that are never closed, one process per page, or a long-lived process that has simply been alive too long. This guide builds a pool that bounds all three, as part of headless browser resource management in the Network Resilience & Proxy Management section.

Problem Framing #

The characteristic symptom is a worker whose resident memory climbs steadily over hours and is eventually killed by the operating system, taking whatever was in flight with it. Per-page measurements look fine — each page uses a reasonable amount — because the growth is in what is never released rather than in what any single page consumes.

The shape of the curve identifies the cause without a profiler.

The three causes are distinguishable by shape. Unclosed contexts produce growth proportional to pages crawled, with a clear step per page. Process-per-page produces high average memory and high processor use with lots of short-lived processes, and it usually shows up first as slow throughput rather than as a memory alert. Process ageing produces slow, roughly linear growth on a stable process count, and it is the one that survives fixing the other two.

A pool that closes contexts deterministically, reuses processes, and retires them on a bounded lifetime addresses all three without needing to identify which one you had.

Step-by-Step Implementation #

1. A lease-based pool #

A lease is retired only when it has no active contexts left.

Model a browser process as a lease with a bounded lifetime. Callers borrow a context, not a browser, and the pool decides when the underlying process is retired.

import asyncio
import time
from contextlib import asynccontextmanager
from dataclasses import dataclass, field

@dataclass
class Lease:
    browser: object
    started_at: float = field(default_factory=time.monotonic)
    contexts_served: int = 0
    active_contexts: int = 0

    def exhausted(self, max_contexts: int, max_age: float) -> bool:
        return (self.contexts_served >= max_contexts
                or time.monotonic() - self.started_at > max_age)

class BrowserPool:
    def __init__(self, playwright, size: int = 2, max_contexts: int = 400,
                 max_age_seconds: float = 1800.0, launch_args: list[str] | None = None):
        self._pw = playwright
        self._size = size
        self._max_contexts = max_contexts
        self._max_age = max_age_seconds
        self._args = launch_args or DEFAULT_ARGS
        self._leases: list[Lease] = []
        self._lock = asyncio.Lock()

    async def start(self) -> None:
        for _ in range(self._size):
            self._leases.append(Lease(await self._launch()))

    async def _launch(self):
        return await self._pw.chromium.launch(headless=True, args=self._args)

    async def _pick(self) -> Lease:
        async with self._lock:
            # Retire any exhausted lease that has no active contexts.
            for index, lease in enumerate(self._leases):
                if lease.exhausted(self._max_contexts, self._max_age) and lease.active_contexts == 0:
                    asyncio.create_task(lease.browser.close())
                    self._leases[index] = Lease(await self._launch())
            return min(self._leases, key=lambda l: l.active_contexts)

    @asynccontextmanager
    async def context(self, **kwargs):
        lease = await self._pick()
        lease.active_contexts += 1
        lease.contexts_served += 1
        ctx = await lease.browser.new_context(**kwargs)
        try:
            yield ctx
        finally:
            await ctx.close()            # deterministic close, even on an exception
            lease.active_contexts -= 1

The finally: await ctx.close() is the single most important line. Contexts closed only on the success path leak on every timeout, every navigation error and every extraction exception — which is to say, precisely on the pages that are already causing trouble.

Retiring only leases with no active contexts avoids killing a browser out from under an in-flight page. The replacement launches immediately so the pool never shrinks.

2. Launch arguments that actually reduce memory #

DEFAULT_ARGS = [
    "--disable-dev-shm-usage",        # use /tmp instead of a small /dev/shm in containers
    "--disable-gpu",
    "--no-sandbox",                   # only where the container already isolates
    "--disable-background-networking",
    "--disable-backgrounding-occluded-windows",
    "--disable-renderer-backgrounding",
    "--disable-features=TranslateUI,BackForwardCache",
    "--js-flags=--max-old-space-size=256",
]

--disable-dev-shm-usage is the one that fixes the most confusing production failure: containers commonly allocate 64 MB to /dev/shm, and a browser that exhausts it crashes with an error that mentions nothing about shared memory. Disabling the back-forward cache matters for a crawler because that cache exists to make user navigation feel fast and a crawler never navigates back.

3. Bound each context as well #

CONTEXT_DEFAULTS = {
    "viewport": {"width": 1280, "height": 900},   # smaller viewport, less raster memory
    "java_script_enabled": True,
    "bypass_csp": False,
    "service_workers": "block",                    # service workers outlive the page
    "ignore_https_errors": False,
}

@asynccontextmanager
async def crawl_context(pool: BrowserPool, user_agent: str):
    async with pool.context(user_agent=user_agent, **CONTEXT_DEFAULTS) as ctx:
        await install_request_filter(ctx)
        ctx.set_default_timeout(15000)
        ctx.set_default_navigation_timeout(20000)
        yield ctx

Blocking service workers is worth calling out. A registered service worker persists beyond the page that registered it, keeps its own cache, and can continue making requests — all of which is invisible in page-level accounting and directly contrary to a crawler’s interest in a clean, bounded footprint.

4. Measure the thing you are bounding #

import os
import psutil

def pool_memory_mb(pool: BrowserPool) -> dict[str, float]:
    """Resident memory of every browser process the pool owns, plus the worker itself."""
    worker = psutil.Process(os.getpid())
    children = worker.children(recursive=True)
    return {
        "worker_mb": worker.memory_info().rss / 1_048_576,
        "browsers_mb": sum(c.memory_info().rss for c in children) / 1_048_576,
        "process_count": len(children),
    }

Export all three as gauges. process_count growing without bound means leases are not being retired; browsers_mb growing on a stable process count means the ageing limit is too generous; worker_mb growing means the leak is in your own code rather than the browser’s.

Verification & Testing #

A memory bound is verified by a soak, not by a unit test. Run the pool over a few hundred representative pages and assert the trend rather than the absolute.

The first invariant catches the leak that only appears on failing pages.

@pytest.mark.asyncio
@pytest.mark.slow
async def test_memory_is_bounded_over_a_soak(pool, sample_urls):
    samples = []
    for index, url in enumerate(sample_urls[:300]):
        async with crawl_context(pool, USER_AGENT) as ctx:
            await render_and_extract(ctx, url, "main")
        if index % 25 == 0:
            samples.append(pool_memory_mb(pool)["browsers_mb"])

    first_quarter = sum(samples[:3]) / 3
    last_quarter = sum(samples[-3:]) / 3
    assert last_quarter < first_quarter * 1.4, f"memory grew: {samples}"

@pytest.mark.asyncio
async def test_context_closes_on_exception(pool):
    before = pool_memory_mb(pool)["process_count"]
    with pytest.raises(RuntimeError):
        async with crawl_context(pool, USER_AGENT):
            raise RuntimeError("extraction failed")
    assert pool_memory_mb(pool)["process_count"] == before

The second test is fast and catches the leak that matters most. A 40% tolerance in the soak accommodates ordinary variance while still failing on genuine unbounded growth.

Compliance & Operational Guardrails #

Every context is closed in a finally, so an extraction failure cannot leak one.
Browser processes are retired on both a context count and an age, before memory becomes a problem.
The crawler token is set per context and matches the token the HTTP stage sends.
Service workers are blocked so no request escapes the crawl’s own accounting.
Requests issued by the rendered page are counted against the host’s rate budget, not just navigations.

Common Mistakes #

Closing contexts only on success. Every failed page leaks, and failures cluster on the sites already causing trouble.
Launching a browser per page. Startup dominates the runtime and memory never stabilises long enough to measure.
Setting no viewport. The default is larger than a crawler needs, and raster memory scales with its area.

Frequently Asked Questions #

How many contexts can one browser process hold? #

Dozens, but the useful limit is memory rather than a hard cap. Two to four concurrently active contexts per process is a reasonable operating point for text extraction; more is fine if the pages are light and the measurements support it. Retire the process after a few hundred cumulative contexts regardless, since the growth is cumulative rather than concurrent.

Should the pool be shared across event loops or processes? #

No. Keep one pool per worker process, sized so the worker’s total memory fits its limit with headroom. Sharing a pool across processes requires marshalling browser handles, which is complexity with no benefit — scaling out means more workers, each with its own small pool.

What about reusing one context across many pages? #

Only when the pages genuinely share a session. Reusing a context across unrelated targets carries cookies and storage between them, which is both a correctness problem and a privacy one. The cost of a fresh context is small; the cost of leaked session state is not.

Should the pool be pre-warmed at start-up? #

Yes. Launching a browser takes hundreds of milliseconds, and a pool that launches lazily pays that cost on the first request of every worker — which is exactly when a crawl is trying to establish its pace. Starting the configured number of processes before accepting work makes the first fetch as fast as the thousandth, and it surfaces a launch failure at start-up rather than mid-crawl.

How should a crashed browser process be handled? #

Detect it on the next context acquisition, replace the lease, and let the affected page’s URL return to the frontier. A crashed browser takes its in-flight contexts with it, so those pages have not been fetched and should be retried like any other transient failure. What must not happen is the pool silently shrinking: a lease that fails to relaunch and is not replaced reduces capacity permanently, and the crawl slows for reasons nobody can find.

Headless Browser Resource Management — deciding whether to render at all before optimising how.
Blocking Images and Fonts to Cut Crawl Bandwidth — the request filter installed on each context here.
Managing Persistent HTTP Sessions — the equivalent lifecycle discipline for plain HTTP clients.