Health Checking and Scoring a Proxy Pool #

An egress pool is only useful if the selector can tell a healthy endpoint from a degraded one, and only ethical if that judgement is based on transport health rather than on which endpoint last avoided a block. This guide builds a scoring system with that separation designed in, as part of building ethical proxy rotation systems in the Network Resilience & Proxy Management section.

Problem Framing #

Endpoints in a pool are not interchangeable. Some are consistently fast, some add 400 ms, some fail one connection in twenty, and some stop working entirely without announcing it. A selector that picks uniformly at random inherits the worst endpoint’s failure rate across the whole crawl; one that picks the fastest exhausts a single endpoint while the rest idle.

Scoring refusals would teach the selector to route around blocks.

The obvious fix — score endpoints by how often requests through them succeed — introduces a subtler problem. If a 403 counts as a failure, the pool learns to prefer endpoints that have not yet been blocked, which is an evasion strategy nobody designed and nobody can defend. The scoring system therefore has to distinguish transport outcomes, which describe the endpoint, from application outcomes, which describe the relationship between your crawl and the target.

Step-by-Step Implementation #

1. Score on transport signals only #

Affinity keeps each target seeing a small, stable set of addresses.

from collections import deque
from dataclasses import dataclass, field
import time

@dataclass
class EndpointHealth:
    """Health of one egress endpoint. Refusals are counted but never scored."""
    endpoint_id: str
    latencies: deque = field(default_factory=lambda: deque(maxlen=64))
    successes: int = 0
    transport_errors: int = 0
    refusals: int = 0                      # recorded for reporting, excluded from score
    last_used: float = 0.0
    quarantined_until: float = 0.0

    def observe(self, outcome: str, latency: float | None, now: float) -> None:
        self.last_used = now
        if outcome == "ok":
            self.successes += 1
            if latency is not None:
                self.latencies.append(latency)
        elif outcome in ("transport_error", "timeout"):
            self.transport_errors += 1
            if self.recent_error_rate() > 0.5:
                self.quarantined_until = now + 300.0
        elif outcome == "refused":
            self.refusals += 1             # the crawl's problem, not the endpoint's

    def recent_error_rate(self) -> float:
        attempts = self.successes + self.transport_errors
        return self.transport_errors / attempts if attempts else 0.0

    def median_latency(self) -> float:
        if not self.latencies:
            return 1.0
        ordered = sorted(self.latencies)
        return ordered[len(ordered) // 2]

    def score(self, now: float) -> float:
        if now < self.quarantined_until:
            return 0.0
        attempts = self.successes + self.transport_errors
        if attempts < 5:
            return 0.5                     # unproven: neither favoured nor punished
        reliability = self.successes / attempts
        return reliability / (1.0 + self.median_latency())

The unproven default of 0.5 gives a new endpoint enough selection pressure to accumulate a sample without letting it dominate before it has proved anything. Quarantining on a high recent error rate takes an obviously broken endpoint out of rotation quickly, and the timed expiry brings it back automatically if the fault was transient.

2. Active health checks for idle endpoints #

An endpoint that has not been used recently has a stale score. A cheap periodic probe against a neutral target — your own endpoint, ideally, so no third party pays for your health checks — keeps the pool’s picture current.

import asyncio
import httpx

HEALTH_URL = "https://health.example.org/ping"     # your own infrastructure
IDLE_THRESHOLD = 600.0

async def probe(endpoint_id: str, proxy_url: str, health: EndpointHealth) -> None:
    started = time.monotonic()
    try:
        async with httpx.AsyncClient(proxy=proxy_url, timeout=10.0) as client:
            resp = await client.get(HEALTH_URL)
        elapsed = time.monotonic() - started
        health.observe("ok" if resp.status_code == 200 else "transport_error",
                       elapsed, time.monotonic())
    except httpx.TransportError:
        health.observe("transport_error", None, time.monotonic())

async def sweep(pool, interval: float = 120.0) -> None:
    while True:
        now = time.monotonic()
        idle = [e for e in pool.endpoints() if now - pool.health(e).last_used > IDLE_THRESHOLD]
        await asyncio.gather(*(probe(e, pool.url(e), pool.health(e)) for e in idle))
        await asyncio.sleep(interval)

Probing against your own endpoint matters. Health-checking against a third-party site means every endpoint in your pool sends a request to somebody else’s infrastructure every two minutes, for your benefit and at their cost — a small unkindness that scales badly with pool size.

3. Select with host affinity, then by score #

import hashlib

def affinity_subset(host: str, endpoints: list[str], width: int = 2) -> list[str]:
    """A stable, small set of endpoints per host — not a fresh pick every request."""
    return sorted(endpoints,
                  key=lambda e: hashlib.sha256(f"{host}:{e}".encode()).digest())[:width]

def select(host: str, pool, now: float) -> tuple[str, str]:
    """Return (endpoint_id, reason)."""
    subset = affinity_subset(host, pool.endpoints())
    healthy = [e for e in subset if pool.health(e).score(now) > 0.3]
    if healthy:
        return max(healthy, key=lambda e: pool.health(e).score(now)), "affinity"
    fallback = max((e for e in pool.endpoints() if e not in subset),
                   key=lambda e: pool.health(e).score(now), default=None)
    if fallback is None:
        raise NoHealthyEndpoint(host)
    return fallback, "affinity_unhealthy"

Affinity is doing real work here beyond stability. A host that sees requests from a stable pair of addresses can allow-list them, can reason about your traffic, and can talk to you about it; a host that sees requests from a different address every time cannot do any of those things, however honest your token is.

4. Report refusals separately and loudly #

def refusal_report(pool, now: float) -> list[dict]:
    """Refusals per endpoint — an operational signal, never a scoring input."""
    return sorted(
        ({"endpoint": e, "refusals": pool.health(e).refusals,
          "successes": pool.health(e).successes} for e in pool.endpoints()),
        key=lambda row: row["refusals"], reverse=True,
    )

If refusals concentrate on particular endpoints, that is worth knowing — it may mean an address range with a poor reputation, or a provider whose addresses appear on a shared blocklist. The correct response is to talk to the provider or retire the range deliberately, as an operational decision recorded by a person. What must never happen is the selector doing it silently.

Verification & Testing #

Probing third-party sites means every endpoint pings someone else on your schedule.

def test_refusals_do_not_affect_the_score():
    h = EndpointHealth("e1")
    for _ in range(20):
        h.observe("ok", 0.2, 0.0)
    before = h.score(0.0)
    for _ in range(50):
        h.observe("refused", None, 0.0)
    assert h.score(0.0) == before

def test_transport_errors_quarantine_quickly():
    h = EndpointHealth("e2")
    for _ in range(6):
        h.observe("transport_error", None, 100.0)
    assert h.score(100.0) == 0.0
    assert h.score(500.0) > 0.0            # quarantine expires

def test_affinity_is_stable_for_a_host():
    endpoints = [f"e{i}" for i in range(20)]
    assert affinity_subset("example.com", endpoints) == affinity_subset("example.com", endpoints)

def test_pool_size_does_not_change_host_rate(simulator):
    assert simulator(pool_size=2, host_rpm=20).requests_in(60) == 20
    assert simulator(pool_size=200, host_rpm=20).requests_in(60) == 20

The first and last tests encode the ethical properties directly, which is the point: they will still be there when the person who wrote the selector has moved on.

Compliance & Operational Guardrails #

Application refusals are recorded and reported but never influence endpoint selection.
Health probes target your own infrastructure, not third-party sites.
Host affinity keeps each target seeing a small, stable set of egress addresses.
Pool size never affects per-host rate; the limiter sits above the pool.
Every selection logs the endpoint identifier and the reason, so egress is explicable.
Endpoint identifiers in logs are stable references, not raw addresses.

Common Mistakes #

Scoring on any non-200. The pool silently learns to route around blocks, which is the behaviour the whole design exists to prevent.
Health-checking against a third-party site. Every endpoint pings someone else’s server on your schedule.
Selecting purely by score. The best endpoint takes all the traffic, ages fastest, and its degradation is hidden by the fact that nothing else has a recent sample.

Frequently Asked Questions #

How wide should the affinity subset be? #

Two is a good default: one primary and one healthy alternative, so an endpoint failure does not force a fallback into the wider pool. Widen to three or four only for hosts you crawl heavily enough that a single endpoint’s connection limits become the constraint.

What score threshold should exclude an endpoint? #

Rather than tuning an absolute threshold, exclude relative to the pool: drop endpoints scoring below about a third of the pool median. That adapts automatically as overall conditions change, whereas a fixed threshold either excludes everything on a bad day or nothing on a good one.

Should scores persist across restarts? #

Keep them in the shared store the rest of the crawl state uses, with a TTL of a few hours. Persisting gives a restarted worker a useful starting picture; expiring keeps a stale judgement from outliving the conditions that produced it.

How should an endpoint be retired permanently? #

Through a person, with a recorded reason, and never automatically in response to refusals. Automatic quarantine on transport errors is correct and temporary; permanent retirement is a contractual and operational decision that belongs in the pool’s configuration. Keeping the two mechanisms distinct is what stops the pool from silently reshaping itself around which addresses happen to be blocked, which is precisely the behaviour the scoring design excludes.

What should happen when the whole pool degrades at once? #

Treat it as a fault on your side until proven otherwise. Simultaneous degradation across unrelated providers is almost never a coincidence — it is usually DNS, an expired certificate bundle, an egress firewall change, or a misconfigured client. The correct response is to stop the crawl rather than to fail over, because failing over spreads a local fault across every remaining endpoint and produces a burst of failed requests at every target you touch.

Building Ethical Proxy Rotation Systems — the boundary between distributing load and disguising it.
Choosing Between Datacenter and Residential Proxies — what belongs in the pool in the first place.
Circuit Breakers for Crawl Workers — the host-level control that a refusal should trigger instead.