Detecting Paywalled Content Programmatically #

A crawler needs to know, per response, whether the content it received is the whole document or a deliberate excerpt. Getting that wrong in one direction fills a dataset with truncated articles labelled as complete; getting it wrong in the other discards perfectly public pages. This guide builds a practical, multi-signal paywall detector and shows how to wire its verdict into the record, as part of respecting paywalls and authentication boundaries within the Compliance & Ethical Crawling Foundations section.

Problem Framing #

Paywalls are implemented in at least four ways, and each leaves a different fingerprint. A hard gate returns a 401 or 402, or redirects to a sign-in URL. A server-side meter returns a truncated document with a subscribe prompt appended. A client-side meter returns the full document and hides it with a script once a cookie counter passes a threshold. And a hybrid returns structured data declaring the document gated while serving a short preview in the body.

The asymmetry argues for a sensitive detector that requires corroboration.

The naive heuristics fail on ordinary pages. “The article is short” also describes a news brief. “The page mentions subscribe” also describes a newsletter promotion in the footer. “The response redirected” also describes a canonicalisation. A detector that fires on any one of these will produce enough false positives that someone disables it, which is the worst possible outcome — an unmonitored pipeline collecting excerpts as if they were articles.

The reliable approach is to score several independent signals and require corroboration, exactly as with challenge detection, and to weight the publisher’s own machine-readable declaration far above any heuristic.

Step-by-Step Implementation #

1. Read the publisher’s declaration first #

A declaration short-circuits the score; nothing else is decisive alone.

Publishers using schema.org markup can state the position directly. isAccessibleForFree: false combined with a hasPart block naming the gated section is an explicit, unambiguous answer that requires no inference.

import json
from lxml import html

def structured_blocks(doc_html: str) -> list[dict]:
    tree = html.fromstring(doc_html)
    blocks = []
    for node in tree.xpath('//script[@type="application/ld+json"]'):
        try:
            payload = json.loads(node.text_content())
        except (json.JSONDecodeError, TypeError):
            continue
        blocks.extend(payload if isinstance(payload, list) else [payload])
    return blocks

def declared_paywall(blocks: list[dict]) -> tuple[bool | None, list[str]]:
    """Return (declared gated?, css selectors of gated regions)."""
    for block in blocks:
        free = block.get("isAccessibleForFree")
        if free is None:
            continue
        gated = str(free).lower() in ("false", "0", "no")
        selectors = []
        for part in block.get("hasPart", []) if isinstance(block.get("hasPart"), list) else []:
            if str(part.get("isAccessibleForFree", "")).lower() == "false":
                selector = part.get("cssSelector")
                if selector:
                    selectors.append(selector)
        return gated, selectors
    return None, []

When the declaration is present, stop: it is authoritative and no heuristic should override it. When it is absent — which is still the common case — fall through to scoring.

2. Score the remaining signals #

from dataclasses import dataclass

GATE_PHRASES = (
    "subscribe to continue reading", "sign in to read the full",
    "this article is for subscribers", "you've reached your limit",
    "create a free account to continue", "members only",
)

@dataclass(frozen=True)
class Score:
    total: float
    signals: tuple[str, ...]

def score_signals(status: int, final_url: str, body_text: str,
                  visible_words: int, median_words: int) -> Score:
    signals, total = [], 0.0

    if status in (401, 402):
        return Score(1.0, ("status-gate",))
    if any(seg in final_url for seg in ("/login", "/signin", "/subscribe", "/register")):
        total += 0.6; signals.append("gate-redirect")

    lowered = body_text.lower()
    if any(phrase in lowered for phrase in GATE_PHRASES):
        total += 0.5; signals.append("gate-copy")
    if median_words and visible_words < median_words * 0.35:
        total += 0.3; signals.append("body-truncated")
    if "paywall" in lowered or "premium-content" in lowered:
        total += 0.2; signals.append("gate-class-name")

    return Score(round(min(total, 1.0), 2), tuple(signals))

A threshold of 0.7 requires two corroborating signals in nearly every combination: gate copy alone (0.5) is not enough, nor is truncation alone (0.3), but together they clear it. Tune against your own corpus rather than adopting the numbers wholesale, and keep the per-signal contributions in the record so a threshold change can be evaluated against historical verdicts.

3. Compare against a per-host baseline #

The truncation signal needs a reference. Use a rolling median of visible word count per host and page type, computed from your own recent crawls, rather than a global constant — a site of one-paragraph briefs and a site of long features have nothing in common here.

from collections import deque

class WordBaseline:
    """Rolling median visible-word count per (host, page_type)."""

    def __init__(self, window: int = 200):
        self.window = window
        self._samples: dict[tuple[str, str], deque] = {}

    def observe(self, host: str, page_type: str, words: int) -> None:
        key = (host, page_type)
        self._samples.setdefault(key, deque(maxlen=self.window)).append(words)

    def median(self, host: str, page_type: str) -> int:
        samples = sorted(self._samples.get((host, page_type), ()))
        return samples[len(samples) // 2] if samples else 0

Only feed the baseline with responses the detector judged ungated, or the median drifts downward as previews accumulate and the detector gradually stops recognising them.

4. Attach the verdict to the record #

def annotate(record: dict, declared: bool | None, score: Score) -> dict:
    if declared is True:
        record.update(access_status="gated", access_confidence=1.0,
                      access_signals=["publisher-declaration"], content_complete=False)
    elif score.total >= 0.7:
        record.update(access_status="gated", access_confidence=score.total,
                      access_signals=list(score.signals), content_complete=False)
    else:
        record.update(access_status="public", access_confidence=1.0 - score.total,
                      access_signals=list(score.signals), content_complete=True)
    return record

Storing the confidence alongside the verdict is what makes a later threshold change re-evaluable without a recrawl.

Verification & Testing #

Build a fixture corpus containing, at minimum: a genuinely public article, a short but complete news brief, a server-truncated preview, a client-side gated page with the full text in the markup, a page carrying isAccessibleForFree: false, and a page whose footer advertises a newsletter subscription. The last two are the discriminating cases — one must be detected on the declaration alone, and the other must not fire despite containing the word “subscribe”.

The third fixture is the one that catches naive phrase matching.

import pytest

CASES = [
    ("public_article.html",     "public"),
    ("short_news_brief.html",   "public"),   # short but complete
    ("newsletter_footer.html",  "public"),   # mentions subscribe, is not gated
    ("server_truncated.html",   "gated"),
    ("client_side_gate.html",   "gated"),
    ("declared_not_free.html",  "gated"),
]

@pytest.mark.parametrize("fixture,expected", CASES)
def test_detector(fixture, expected, baseline):
    body = load_fixture(fixture)
    declared, _ = declared_paywall(structured_blocks(body))
    score = score_signals(200, "https://example.com/a", visible_text(body),
                          count_words(body), baseline.median("example.com", "article"))
    record = annotate({}, declared, score)
    assert record["access_status"] == expected

In production, watch two rates: the proportion of responses judged gated per host, and the proportion judged gated with a confidence between 0.6 and 0.8. A rising share in that middle band means the detector is operating near its threshold on that host and the fixture corpus needs a fresh sample.

Compliance & Operational Guardrails #

A gated verdict must reduce what is stored, never trigger a retry or an identity change.
Preview content is stored with content_complete: false; the gated remainder is never reconstructed.
The publisher’s isAccessibleForFree declaration overrides every heuristic, in both directions.
Gate verdicts are re-evaluated on every crawl, never cached from an earlier generation.
Confidence and contributing signals are stored so a threshold change is auditable.

Common Mistakes #

Using a global word-count threshold. It fires constantly on sites with short articles and never on sites with long ones.
Feeding gated responses into the baseline. The median drifts toward the preview length and the detector goes blind.
Letting a gated verdict trigger the retry path. The crawler repeatedly requests content it has been refused, which is exactly the pattern to avoid.

Frequently Asked Questions #

Should the detector run before or after extraction? #

After parsing but before the write. It needs the visible text and any structured-data block, both of which come from the parse, and its verdict determines what the write stage stores. Running it earlier means duplicating the parse; running it later means the record has already been written as complete.

What if a site declares content free but visibly truncates it? #

Trust the observed truncation and record the conflict. A stale or incorrect declaration is a site bug, and storing a preview marked complete is the outcome you most want to avoid. Log the disagreement — a host where declaration and heuristic diverge consistently is worth a manual look.

How often should the fixture corpus be refreshed? #

Quarterly for stable publishers, and immediately whenever the gated-rate metric for a host shifts sharply. Paywall implementations change with marketing campaigns more often than with engineering releases, so the signal that a refresh is due usually comes from the metric rather than from a site announcement.

How should the detector treat a page that is gated for some regions only? #

As gated, and record the region the request originated from. Geographic gating is common for licensed content, and a crawl running from several regions will see different verdicts for the same URL. Storing the observing region alongside the verdict makes that explicable rather than looking like a flapping detector, and it also surfaces a question worth asking: if the content is licensed by territory, crawling it from a region where it is available in order to serve users elsewhere is a licensing question rather than a technical one.

What if a site gates only part of a page? #

That is the common case for metered publishers, and the structured hasPart declaration is designed for exactly it: the publisher names the CSS selector of the gated region. Where that declaration exists, extract everything outside the named region and mark the record incomplete. Where it does not, treat the visible portion as the preview and record that the boundary was inferred rather than declared, so a later review can tell the two apart.

Respecting Paywalls and Authentication Boundaries — the policy this detector implements.
Handling Login Walls Without Credential Sharing — what to do once a hard gate is confirmed.
Parsing Schema.org Product Data From JSON-LD — reading the structured block this detector depends on.