Compliant Web Scraping

Choosing Between Datacenter and Residential Proxies #

The choice between egress types is usually framed as a technical trade-off — cost against block rate — and framed that way it leads teams to the wrong answer. The decisive question is whether the addresses you are routing through were obtained with the informed consent of the people whose connections they belong to. This guide sets out how to evaluate that, and what a defensible egress strategy looks like, as part of building ethical proxy rotation systems in the Network Resilience & Proxy Management section.

Problem Framing #

Four categories of egress capacity are sold, and they differ in consent provenance far more than in throughput.

Egress tiers by consent provenanceMove down this list only when a recorded requirement forces it.Egress tiers by consent provenanceDatacenterone contract, fully verifiable, the defaultISP-hostedone contract, consumer-registered rangeResidentialconsent runs through the provider’s end usersMobilehardest to verify, and it spends someone’s data
Move down this list only when a recorded requirement forces it.

Datacenter addresses belong to a hosting provider you contract with directly. The consent chain is one link long and completely verifiable. They are cheap, fast, and recognisable as datacenter traffic — which is only a problem if being recognisable as automated is a problem, and for a compliant crawler it is not.

ISP-hosted addresses are datacenter machines using address ranges registered to a consumer ISP. Provenance is still clear, and they appear to targets as ordinary connections. They cost more and are the reasonable step up when a target’s infrastructure genuinely mishandles datacenter ranges.

Residential addresses route through real people’s home connections. The consent chain runs through whatever agreement the provider has with those people, and that agreement is frequently a checkbox in the installer of an unrelated free application. This is where the ethical question becomes concrete.

Mobile addresses route through cellular connections and carry the same consent question in a form that is even harder to verify, alongside the fact that the traffic consumes somebody’s data allowance.

The technical differences are real but secondary. What separates a defensible egress strategy from an indefensible one is whether you can explain, to the person whose connection carried your request, how they came to agree to that.

Step-by-Step Implementation #

1. Start with the cheapest defensible option and justify moving #

Diagnose the block before changing tierMost blocks resolve above the last line and never need a tier change.Diagnose the block before changing tier✓Rate is within the published or agreed budget✓Concurrency is at or below two connections✓Every request carries the token and a working contact URL✓The path is permitted by the current rules snapshot✓Only then: is the address range itself the issue
Most blocks resolve above the last line and never need a tier change.
from dataclasses import dataclass

@dataclass(frozen=True)
class EgressTier:
    name: str
    consent_chain: str
    requires_audit: bool
    justification_required: bool

TIERS = (
    EgressTier("datacenter", "direct contract with the hosting provider", False, False),
    EgressTier("isp",        "direct contract; ISP-registered range",     False, True),
    EgressTier("residential","provider's agreement with end users",       True,  True),
    EgressTier("mobile",     "provider's agreement with device owners",   True,  True),
)

def assert_tier_permitted(host_entry) -> None:
    tier = next(t for t in TIERS if t.name == host_entry.egress_tier)
    if tier.justification_required and not host_entry.egress_justification:
        raise ConfigError(f"{host_entry.host}: {tier.name} egress needs a recorded justification")
    if tier.requires_audit and not host_entry.provider_audit_ref:
        raise ConfigError(f"{host_entry.host}: {tier.name} egress needs a completed provider audit")

Enforcing this in configuration validation rather than in a policy document is what makes it stick. A host entry that specifies residential egress without an audit reference fails the build, and the discussion happens before the crawl runs rather than after.

2. Audit the provider before residential capacity enters the pool #

Five questions, answered in writing, with the answers stored alongside the contract:

  • How is consent obtained from the people whose connections are used? A specific, prominent disclosure is a different thing from a clause in a bundled installer.
  • How can a participant see that their connection is being used, and opt out? A route that exists and works, not one that exists in principle.
  • What traffic is permitted through the network, and how is that enforced? A provider that cannot describe its enforcement is describing an unenforced policy.
  • Is your specific use permitted by the contract? Many providers prohibit categories of use that a buyer would assume were fine.
  • What is logged about the participants and about your traffic, by whom, for how long?

A provider unable to answer these clearly is a provider whose capacity you cannot defend using. That is a sufficient reason to decline it, independent of price or block rate.

3. Check whether you actually need it #

Teams frequently reach for residential capacity because a target returns a block, without first checking whether the block was about the address at all. In practice most such blocks are about rate, concurrency, or an unidentified client — all of which are fixable without changing egress, and none of which are fixed by changing egress for more than a few hours.

BLOCK_DIAGNOSTICS = [
    ("rate", "Is per-host rate within the published or agreed budget?"),
    ("concurrency", "Is per-host concurrency at or below two connections?"),
    ("identity", "Does every request carry the crawler token and a working contact URL?"),
    ("rules", "Is the path permitted by the current rules snapshot?"),
    ("terms", "Has the terms document changed since the last review?"),
    ("egress", "Only after ALL of the above: is the address range itself the issue?"),
]

Working through that list in order resolves the great majority of blocks without touching the pool. Reaching the last item and finding the answer is genuinely yes — a shared datacenter range with a poor reputation, for instance — is a real finding, and ISP-tier capacity is the proportionate response to it.

4. Record the decision where it can be reviewed #

# registry/marketplace.example.com.yaml (excerpt)
egress:
  tier: isp
  provider: example-transit
  contract_ref: CTR-2026-118
  justification: >
    Datacenter ranges receive a 503 from this host's edge configuration on every
    request, confirmed against three unrelated providers on 2026-05-02. Rate,
    concurrency and identification were verified compliant first (see incident
    INC-2026-2210). ISP-tier capacity resolves it; residential was not required.
  reviewed_by: data-governance
  reviewed_on: 2026-05-06
  review_due: 2026-11-06

The justification records that the cheaper diagnoses were excluded first. Six months later, when someone asks why this host uses a more expensive tier, that paragraph is the answer — and the review date forces the question of whether it is still true.

Verification & Testing #

Provider audit questions and what a poor answer looks likeA provider that cannot answer these is one you cannot defend using.Provider audit questions and what a poor answer looks likeQuestionGood answerPoor answerHow is consent obtainedSpecific disclosureBundled installerHow does a user opt outA working routeOn requestIs our use permittedNamed in contractNot prohibitedWhat is logged, and for how longDocumentedUnclear
A provider that cannot answer these is one you cannot defend using.
def test_residential_requires_an_audit(registry):
    for entry in registry:
        if entry.egress_tier in ("residential", "mobile"):
            assert entry.provider_audit_ref, f"{entry.host}: no provider audit on record"
            assert entry.egress_justification, f"{entry.host}: no justification recorded"

def test_egress_tier_is_not_widened_by_a_block(crawler):
    crawler.inject(host="example.com", status=403, names_crawler=True)
    assert crawler.egress_tier("example.com") == "datacenter"   # unchanged
    assert crawler.host_state("example.com") == "paused"

def test_all_egress_is_attributable(pool):
    for endpoint in pool.endpoints():
        assert pool.provider_of(endpoint) is not None
        assert pool.contract_ref(endpoint) is not None

The middle test encodes the rule that matters most: a refusal pauses the host and does not escalate the egress tier. Making that a test rather than a convention is what keeps it true through the next refactor.

Compliance & Operational Guardrails #

  • Datacenter egress is the default; every step up is justified in writing and reviewed on a schedule.
  • Residential and mobile capacity requires a completed provider audit covering consent and opt-out.
  • Every endpoint in the pool traces to a provider, a contract, and a documented consent basis.
  • A block never triggers an automatic change of egress tier.
  • Egress justifications expire and are re-examined, exactly like a rules snapshot or an assessment.

Common Mistakes #

  1. Treating block rate as the selection criterion. It optimises directly for evasion and skips the question that actually matters.
  2. Buying residential capacity without auditing the provider. The consent chain is the whole risk, and it is invisible from the product page.
  3. Escalating tier as an incident response. It converts a fixable rate or identification problem into a durable ethical one.

Frequently Asked Questions #

Is residential egress ever appropriate? #

Occasionally — geographic verification of content genuinely served differently by region is the clearest example, where no datacenter presence exists in that region. Even then the burden is a provider whose consent chain you have audited, a recorded justification, and a scope limited to the specific need rather than a general upgrade of the pool.

What if a target blocks all datacenter ranges outright? #

That is information about what the target wants. Some sites block datacenter traffic as a deliberate anti-automation posture, in which case routing around it via consumer addresses is defeating a control rather than solving a technical problem. Ask the operator whether an allow-list or an API is available; where the block is an over-broad edge configuration rather than an intent, they will usually say so.

Does using ISP-tier capacity count as disguising the crawler? #

Not if the crawler still identifies itself. The token, the contact URL and the honest rate are what make a crawler identifiable; the address range is not an identity claim. What would cross the line is combining consumer addresses with a browser-impersonating token and no contact route — at that point every signal is pointed away from the truth.

How should an existing residential pool be wound down? #

Deliberately and with measurement, not overnight. Move the hosts that never needed it back to datacenter capacity first, confirm that success rates hold, and let the remaining set shrink to the cases with a recorded justification. In most pools that set turns out to be a handful of hosts rather than the majority, and the cost saving is substantial. Where a host genuinely requires it, the audit and the justification make the remaining usage defensible instead of ambient.

Does egress choice affect what the crawl is allowed to collect? #

Not directly, but it affects what you can demonstrate. A crawl whose requests are attributable to a contract and a range is one whose logs can be reconciled with a target’s; one routed through rotating consumer addresses cannot be reconciled with anything. If a site operator ever asks which of their pages you fetched and from where, the answer needs to exist — and that is a property of the egress arrangement rather than of the crawler code.