Connection Pool Tuning for httpx and requests #

Default connection-pool settings are tuned for an application making occasional outbound calls, not for a crawler making thousands. Left alone they produce symptoms that look like network faults — intermittent resets, latency that climbs with concurrency, sockets accumulating in CLOSE_WAIT — and teams spend days chasing the target before finding the client. This guide sets the values that matter, as part of managing persistent HTTP sessions in the Network Resilience & Proxy Management section.

Problem Framing #

Three defaults cause nearly all the trouble.

Each is a one-line fix and a day of investigation if left alone.

Pool size per host. requests defaults to ten connections per host and, unless configured otherwise, discards the excess rather than queueing — with a warning that is easy to miss. httpx defaults are more generous but still unrelated to whatever per-host concurrency you actually intended.

Keep-alive expiry. Clients hold idle connections longer than many servers will. When the server closes an idle connection and the client reuses it anyway, the next request fails with a reset — reported by your classifier as a transient host error, when the host did nothing wrong.

Pool acquisition timeout. Frequently unset, which means a saturated pool queues without bound. Latency climbs, nothing times out, and the symptom presents as a slow target rather than an overloaded client.

All three are one-line fixes once you know they exist.

Step-by-Step Implementation #

1. Derive pool size from your intended concurrency #

The pool ceiling follows from politeness, never the other way round.

The pool should never be larger than the concurrency you have decided is polite. If per-host concurrency is two, a pool permitting ten connections to that host means a burst can open ten — a footprint you did not choose.

import httpx

PER_HOST_CONCURRENCY = 2
DISTINCT_HOSTS_IN_FLIGHT = 20

limits = httpx.Limits(
    max_connections=PER_HOST_CONCURRENCY * DISTINCT_HOSTS_IN_FLIGHT,   # process-wide ceiling
    max_keepalive_connections=PER_HOST_CONCURRENCY * DISTINCT_HOSTS_IN_FLIGHT // 2,
    keepalive_expiry=20.0,             # comfortably below typical server idle timeouts
)

timeout = httpx.Timeout(
    connect=5.0,      # a host that cannot handshake in 5 s will not serve a page
    read=20.0,        # time between response chunks
    write=10.0,
    pool=5.0,         # how long to wait for a free connection — never leave this unset
)

client = httpx.Client(
    limits=limits,
    timeout=timeout,
    headers=CRAWLER_HEADERS,
    follow_redirects=False,            # redirects are a crawl decision, not a transport one
    http2=True,
)

httpx has no per-host limit, so the per-host bound has to come from the semaphore described in per-host concurrency limits with asyncio semaphores. The pool ceiling then bounds the process, and the semaphore bounds each target.

2. The same settings in `requests` #

import requests
from requests.adapters import HTTPAdapter

def build_session(per_host: int = 2, hosts_in_flight: int = 20) -> requests.Session:
    session = requests.Session()
    adapter = HTTPAdapter(
        pool_connections=hosts_in_flight,   # number of host pools to keep
        pool_maxsize=per_host,              # connections per host pool
        pool_block=True,                    # QUEUE when saturated, do not discard
        max_retries=0,                      # retries belong in the crawler, not the adapter
    )
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    session.headers.update(CRAWLER_HEADERS)
    return session

pool_block=True is the important one. The default is False, which means a request arriving at a saturated pool opens a new connection outside the pool and discards it afterwards — silently exceeding the concurrency you configured, and logging only a warning. With blocking enabled the pool is a real bound.

Setting max_retries=0 keeps retry policy in one place. Adapter-level retries are invisible to your classifier, do not respect your budget, and will happily retry a refusal.

3. Keep keep-alive below the server’s idle timeout #

Common server idle timeouts sit between 5 and 75 seconds; nginx defaults to 75, Apache to 5, and many CDNs to around 30. A client expiry of 20 seconds is below all the common values, so the client discards an idle connection before the server does and the reset never happens.

def diagnose_reset_rate(metrics) -> str | None:
    """A reset rate that scales with idle time is a keep-alive mismatch, not a host fault."""
    if metrics.resets_after_idle_over_20s > metrics.resets_after_idle_under_5s * 3:
        return ("connection resets cluster on connections idle >20 s — "
                "reduce keepalive_expiry below the server's idle timeout")
    return None

Recording idle time at the point of reuse makes this diagnosable in minutes rather than days. Without it, the resets look random.

4. Close responses, always #

Pool exhaustion in a crawler is almost always a leak: a response whose body was never read or closed keeps its connection checked out forever.

from contextlib import contextmanager

@contextmanager
def fetch(client: httpx.Client, url: str):
    """Guarantees the connection returns to the pool, even on an exception."""
    response = client.send(client.build_request("GET", url), stream=True)
    try:
        yield response
    finally:
        response.close()

# Streaming, with a hard size ceiling so a huge body cannot exhaust memory.
MAX_BYTES = 25 * 1024 * 1024

def read_bounded(response: httpx.Response) -> bytes:
    chunks, total = [], 0
    for chunk in response.iter_bytes():
        total += len(chunk)
        if total > MAX_BYTES:
            raise ValueError(f"response exceeded {MAX_BYTES} bytes")
        chunks.append(chunk)
    return b"".join(chunks)

The size ceiling belongs here rather than in the parser: refusing an oversized body at the transport layer avoids buffering it at all, and a worker killed by an unexpectedly large response takes every in-flight request with it.

Verification & Testing #

Reuse is a politeness measure as much as a performance one.

def test_pool_blocks_rather_than_exceeding(session):
    """With pool_block=True, concurrency never exceeds pool_maxsize."""
    peak = run_concurrent(session, url="https://slow.test/x", tasks=20, observe_peak=True)
    assert peak <= 2

def test_pool_timeout_surfaces_saturation(client):
    with pytest.raises(httpx.PoolTimeout):
        run_concurrent(client, url="https://slow.test/x", tasks=200, timeout_after=6.0)

def test_response_is_closed_on_exception(client, pool_stats):
    before = pool_stats.checked_out()
    with pytest.raises(RuntimeError):
        with fetch(client, "https://example.test/x"):
            raise RuntimeError("extraction failed")
    assert pool_stats.checked_out() == before

Operationally, export three gauges: connections checked out, pool wait time, and reset rate broken down by idle duration. Sustained pool wait means the pool is undersized for the concurrency you are attempting, which is a scheduling problem rather than a pool problem — the answer is fewer concurrent hosts, not a bigger pool.

Compliance & Operational Guardrails #

Pool size per host never exceeds the concurrency the host budget permits.
The pool blocks when saturated; it never silently opens connections outside its bound.
Redirects are handled by the crawler so authorisation and rules checks re-run on the target.
Response bodies are closed in a finally and bounded by an explicit size ceiling.
Adapter-level retries are disabled so the crawler’s classifier sees every outcome.

Common Mistakes #

Leaving pool_block=False in requests. Concurrency silently exceeds the configured maximum, with only a warning to show for it.
Leaving the pool timeout unset. Saturation becomes unbounded queueing that presents as a slow target.
Following redirects at the transport layer. A redirect can move a request from an allowed path to a disallowed one below the layer that checks.

Frequently Asked Questions #

Should HTTP/2 be enabled? #

Yes where the target supports it. Multiplexing reduces the connection count substantially, which is easier on both sides. Note that concurrency accounting has to change with it: count in-flight streams, not connections, because the origin’s work scales with requests regardless of how they are framed.

How do I size the process-wide ceiling? #

From the number of distinct hosts you expect in flight, multiplied by per-host concurrency, plus a small margin. Then check it against the process file-descriptor limit — a ceiling above the descriptor limit produces failures that look like network errors and are not.

Does a proxy change the pool arithmetic? #

Yes. Connections are pooled per proxy-and-target pair, so a pool of endpoints multiplies the number of distinct pools your process maintains. Keep the ceiling in terms of total connections rather than per-host, and remember that host affinity — a small stable endpoint subset per host — keeps that multiplication modest, as covered in building ethical proxy rotation systems.

Does a single client instance serve the whole crawl? #

One per worker process is right, shared across all hosts that worker touches. Creating a client per request discards the pool entirely and turns every fetch into a fresh handshake, which is the failure this whole layer exists to prevent. Creating one per host multiplies the process-wide ceiling by the host count and makes the total connection footprint impossible to reason about. One client, one set of limits, per-host bounds enforced above it.

How should the pool interact with a crawl that pauses hosts? #

A paused host simply stops receiving requests, and its idle connections expire naturally through the keep-alive timeout. There is no need to close them explicitly, and doing so is usually counterproductive — a host paused for five minutes will be crawled again shortly, and discarding the connections means paying for fresh handshakes when it resumes. Let the expiry handle it.

What should be monitored to know the pool is right? #

Three gauges: connections currently checked out, time spent waiting for a free connection, and the reset rate broken down by how long the connection had been idle. Checked-out connections approaching the ceiling means the process is at capacity; sustained pool wait means the concurrency being attempted exceeds what the pool allows; and resets clustering on long-idle connections is the keep-alive mismatch described above. Together they distinguish a client problem from a host problem, which is the distinction that otherwise costs a day of investigation.

Managing Persistent HTTP Sessions — session lifecycle, cookie scoping, and where the pool sits.
Per-Host Concurrency Limits With Asyncio Semaphores — the per-host bound the pool ceiling complements.
Exponential Backoff and Retry Logic — where retries belong once the adapter stops doing them.