Handling Login Walls Without Credential Sharing

Once a crawler confirms that content sits behind a login, the engineering question stops being “how do we get in” and becomes “what are we authorised to do, and how do we implement that safely”. This guide covers the routes that exist when content is genuinely gated, and the credential-handling practices that apply when an authorised route does involve an account. It belongs to respecting paywalls and authentication boundaries in the Compliance & Ethical Crawling Foundations section.

Problem Framing #

The situation arises constantly: a research project needs archived filings behind a member login, a price-monitoring pipeline needs a trade catalogue available only to registered buyers, an internal tool needs data from a platform where the company already has a paid seat. In each case somebody holds valid credentials, and the shortest path is to paste them into the crawler’s configuration.

Work down the list before considering any authenticated crawl.

That shortest path fails on three counts. The account terms almost always prohibit automated access and sharing, so the fastest outcome is a terminated account. The credentials become a secret sprawled across a worker fleet, a configuration store and probably a chat message. And the crawl becomes attributable to an individual rather than to an organisation, which is an unfair position to put a colleague in when something goes wrong.

The alternative is not “give up”. It is to establish an authorised route, and then to implement it with the same care as any other production credential.

Step-by-Step Implementation #

1. Look for a sanctioned interface before anything else #

Every record traces back to the authorisation that permitted it.

Most platforms that gate content also publish a route for programmatic access, and it is frequently cheaper and more reliable than crawling. Check, in this order: a documented API with its own terms; a bulk export or data-dump offering; a research or academic access programme; a partner or reseller agreement; and finally a direct request to the operator. The contact route your crawler identification publishes is the natural channel for the last of these, and an email describing what you want, why, and at what volume is answered more often than teams expect.

Record the outcome in the host’s registry entry whichever way it goes. A documented “asked, refused, do not crawl” is as valuable as a permission, because it stops the question being reopened every quarter by someone who does not know it was asked.

2. Where an account is authorised, use a service identity #

If access is granted, obtain an identity that belongs to the organisation rather than to a person: a service account, an API client, or a seat provisioned for the integration. This matters for revocation as much as for propriety — a person leaving should not break the pipeline, and a compromised pipeline credential should not affect a person’s own access.

import os
import httpx

class AuthorisedClient:
    """Wraps an authorised session. Credentials come from the secret store only."""

    def __init__(self, secret_ref: str, token_url: str, agreement_ref: str):
        self.agreement_ref = agreement_ref          # links every request to its authorisation
        self._secret_ref = secret_ref
        self._token_url = token_url
        self._token: str | None = None
        self._client = httpx.Client(headers=CRAWLER_HEADERS, timeout=30.0)

    def _fetch_token(self) -> str:
        creds = secrets.get(self._secret_ref)        # never from env, never from a file in the repo
        resp = self._client.post(self._token_url, json={
            "client_id": creds["client_id"],
            "client_secret": creds["client_secret"],
            "grant_type": "client_credentials",
        })
        resp.raise_for_status()
        return resp.json()["access_token"]

    def get(self, url: str) -> httpx.Response:
        if self._token is None:
            self._token = self._fetch_token()
        resp = self._client.get(url, headers={"Authorization": f"Bearer {self._token}"})
        if resp.status_code == 401:
            self._token = self._fetch_token()        # single refresh, then give up
            resp = self._client.get(url, headers={"Authorization": f"Bearer {self._token}"})
        return resp

Two details are deliberate. A 401 triggers exactly one refresh, not a retry loop — repeated authentication failures against a gated endpoint look like credential stuffing in the site’s logs, whatever the intent. And agreement_ref travels with the client so every record it produces can be traced to the authorisation that permitted it, which is the same lineage discipline described in documenting scraping authorization and data lineage.

3. Keep the session scoped and observable #

An authorised session should behave more conservatively than an anonymous crawl, not less. The account is attributable, the terms are specific, and the operator is watching.

AUTHORISED_LIMITS = {
    "max_concurrency": 1,            # one session, one request at a time
    "max_requests_per_minute": 10,   # or whatever the agreement states, whichever is lower
    "allowed_path_prefixes": ("/archive/", "/filings/"),
    "session_max_age_seconds": 3600,
}

def assert_in_scope(url_path: str) -> None:
    if not url_path.startswith(AUTHORISED_LIMITS["allowed_path_prefixes"]):
        raise NotAuthorised(f"{url_path} is outside the agreed scope")

A single concurrent session is the right default for an authenticated crawl. Concurrent sessions from one account are the clearest possible signal of automation to a platform’s abuse detection, and they are frequently prohibited outright by the terms attached to a seat.

4. Never store the credential in the dataset #

Session tokens leak into datasets through three routes: a logged request URL containing a token parameter, a stored response body containing a personalised block, and a saved browser storage state committed alongside fixtures. Redact at the logging layer, strip personalisation from stored records, and keep storage-state files out of the repository entirely — the practices set out in session cookie management in headless browsers.

Verification & Testing #

Assert the guardrails rather than assuming them. Three tests catch the failures that matter:

The last rule keeps a repeated failure from resembling credential stuffing.

def test_credentials_never_enter_the_record(authorised_client, sink):
    record = crawl_one(authorised_client, "/archive/2026/filing-1")
    serialised = json.dumps(record)
    assert "Bearer" not in serialised
    assert secrets.get(SECRET_REF)["client_secret"] not in serialised

def test_out_of_scope_path_is_refused(authorised_client):
    with pytest.raises(NotAuthorised):
        crawl_one(authorised_client, "/admin/users")

def test_401_triggers_one_refresh_not_a_loop(mock_transport):
    mock_transport.script = [(401, "", {}), (401, "", {})]
    with pytest.raises(AuthenticationError):
        AuthorisedClient(...).get("https://example.com/archive/x")
    assert mock_transport.token_requests == 1

Operationally, alert on the count of authentication failures per hour. A healthy authorised crawl produces approximately zero; anything above that is either an expired credential or a scope problem, and both want a person rather than a retry.

Compliance & Operational Guardrails #

Credentials belong to the organisation, live in a secret store, and are never committed or logged.
The agreement reference is recorded on every request and every stored record.
Concurrency is one session; request rate is the lower of the agreement’s limit and your own floor.
Path scope is enforced in code, so a discovery pass cannot widen the crawl beyond the agreement.
A 401 refreshes once, then escalates to a human — never a retry loop.
When the agreement expires, the registry entry expires with it and the crawl stops automatically.

Common Mistakes #

Using a colleague’s personal login. It breaches the account terms, exposes an individual, and breaks the moment they change their password.
Running several authenticated workers in parallel. Concurrent sessions from one seat are usually prohibited and always conspicuous.
Letting link discovery pull the authenticated session outside the agreed paths. Scope has to be enforced at request time, not assumed from the seed list.

Frequently Asked Questions #

Not as a way around the wall. Creating an account accepts the site’s terms, and those terms almost always prohibit the automated collection the account is being created for — so the act of accepting them is the act of breaching them. Where an account is genuinely the intended route for your use, the operator will say so, and that conversation is worth having explicitly.

What if the agreement permits access but says nothing about rate? #

Apply your own floor and tell the operator what it is. An authorised crawl that behaves conservatively rarely causes friction; one that treats silence on rate as permission for unlimited concurrency reliably does. The polite rate limiting defaults are a sound starting point, tightened rather than relaxed for authenticated sessions.

How should credentials be rotated? #

On a schedule defined with the operator, through the secret store, with the pipeline reading the current value at session start rather than at deploy time. Rotation should never require a code change or a redeploy — if it does, it will not happen, and a credential that is never rotated is the one that turns up in an old log a year later.

What if the operator never replies to an access request? #

Record the attempt, the date, and the channel used, then treat the answer as no. Silence is not permission, and a documented “asked on this date, no response within four weeks, proceeding without the gated content” is a defensible position that also stops the question being reopened every quarter. Where the data matters enough, a second approach through a different channel — a published partnerships address rather than a general enquiry form — is often more productive than a follow-up on the first.

How should authorised and unauthorised crawls of the same host be kept apart? #

Different registry entries, different scopes, and ideally different workers. Mixing an authorised session with anonymous crawling of the same host makes it very easy for a discovery pass to pull authenticated requests into paths the agreement never covered. Keeping them separate means the authorised path has its own narrow allow-list and its own rate, and the anonymous path cannot inherit the session by accident.

Respecting Paywalls and Authentication Boundaries — the boundary rules this guide operationalises.
Detecting Paywalled Content Programmatically — confirming a gate before deciding what to do about it.
Session Cookie Management in Headless Browsers — keeping authenticated session state contained and verifiable.

Handling Login Walls Without Credential Sharing #