Compliant Web Scraping

Falling Back From JSON-LD to Microdata #

Pages routinely carry structured data in more than one format, and the copies do not always agree. Deciding which wins — and what to do when they conflict — is the difference between an extractor that quietly picks whichever it happens to read first and one whose output you can explain. This guide sets out a precedence chain across JSON-LD, microdata, RDFa and selectors, as part of extracting JSON-LD and microdata in the Data Parsing & Transformation Pipelines section.

Problem Framing #

Multiple formats coexist for mundane reasons. A platform emits microdata in its templates while a marketing plugin adds JSON-LD; a site migrates from one to the other and leaves both in place; a theme includes RDFa that nobody has touched in years. The result is that a single page may state a price three times, and the values can differ because one of the sources is stale.

Default precedence, per fieldFree text inverts this order: the rendered page is what a reader sees.Default precedence, per fieldJSON-LDusually generated by current toolingMicrodataembedded in templates that outlive redesignsRDFararer, occasionally the only complete sourceSelectorsinference, and the last resort
Free text inverts this order: the rendered page is what a reader sees.

Field-level completeness varies too. A JSON-LD block may carry the price and omit the availability that the microdata has; taking the block that “wins” wholesale discards data that was available. What you want is precedence per field, not per format — with the source of each value recorded so a conflict can be investigated.

Step-by-Step Implementation #

1. Extract every source independently #

Assembling one record from several sourcesNormalising first stops formatting differences reading as conflicts.Assembling one record from several sources1Gather candidatesfrom every source2Normalise beforecomparing3Resolve per fieldby precedence4Flag or quarantineon conflict
Normalising first stops formatting differences reading as conflicts.
from dataclasses import dataclass, field

@dataclass
class FieldValue:
    value: object
    source: str            # "json-ld" | "microdata" | "rdfa" | "selector"
    confidence: float

@dataclass
class Candidates:
    """All values found for a field, from every source."""
    by_field: dict[str, list[FieldValue]] = field(default_factory=dict)

    def add(self, name: str, value, source: str, confidence: float) -> None:
        if value in (None, "", []):
            return
        self.by_field.setdefault(name, []).append(FieldValue(value, source, confidence))

def gather(document_html: str, tree, site_config) -> Candidates:
    found = Candidates()

    nodes = json_ld_nodes(document_html)
    product = primary_product(nodes, build_index(nodes), site_config.canonical_url)
    if product:
        for name, value in product_from_json_ld(product).items():
            found.add(name, value, "json-ld", 0.95)

    for item in microdata_items(tree):
        for name, value in map_microdata(item).items():
            found.add(name, value, "microdata", 0.85)

    for item in rdfa_items(tree):
        for name, value in map_rdfa(item).items():
            found.add(name, value, "rdfa", 0.80)

    for name, (value, rule) in extract_with_selectors(tree, site_config).items():
        found.add(name, value, "selector", 0.60 if rule == "primary" else 0.40)

    return found

Gathering everything before choosing is what makes per-field precedence possible, and it costs almost nothing — the parse has already happened, and each extractor is a walk over a tree already in memory.

2. Resolve per field, with conflict detection #

SOURCE_RANK = {"json-ld": 4, "microdata": 3, "rdfa": 2, "selector": 1}

# Free text reads better from the rendered page than from a templated block.
PREFER_VISIBLE = {"description", "body_text"}

def resolve_field(name: str, candidates: list[FieldValue]) -> tuple[object, str, bool]:
    """Return (value, source, conflicted)."""
    if not candidates:
        return None, "none", False

    if name in PREFER_VISIBLE:
        ordered = sorted(candidates, key=lambda c: (c.source == "selector", c.confidence),
                         reverse=True)
    else:
        ordered = sorted(candidates, key=lambda c: (SOURCE_RANK[c.source], c.confidence),
                         reverse=True)

    winner = ordered[0]
    distinct = {normalise_for_comparison(c.value) for c in candidates}
    return winner.value, winner.source, len(distinct) > 1

def resolve_record(found: Candidates) -> tuple[dict, dict, list[str]]:
    record, sources, conflicts = {}, {}, []
    for name, candidates in found.by_field.items():
        value, source, conflicted = resolve_field(name, candidates)
        record[name] = value
        sources[name] = source
        if conflicted:
            conflicts.append(name)
    return record, sources, conflicts

The PREFER_VISIBLE exception is worth keeping. Structured descriptions are frequently truncated to a marketing summary, whereas the rendered page carries what a reader actually sees — so for free text the ranking inverts.

3. Normalise before comparing, or every field looks conflicted #

from decimal import Decimal, InvalidOperation

def normalise_for_comparison(value):
    """Compare meanings, not spellings."""
    if isinstance(value, str):
        text = " ".join(value.split()).strip().lower()
        try:
            return Decimal(text.replace(",", ""))     # "1299.00" == "1299"
        except InvalidOperation:
            return text
    if isinstance(value, (int, float, Decimal)):
        return Decimal(str(value))
    return value

Without this, "1299.00" from JSON-LD and "1,299" from a selector register as a conflict on every single record, the conflict rate sits at 100%, and the signal is discarded as noise within a week.

4. Record sources and act on conflicts #

def finalise(record: dict, sources: dict, conflicts: list[str], context: dict) -> dict:
    record["field_sources"] = sources
    record["structured_coverage"] = sum(
        1 for s in sources.values() if s in ("json-ld", "microdata", "rdfa")
    ) / max(len(sources), 1)

    for name in conflicts:
        metrics.field_conflicts.labels(field=name, host=context["host"]).inc()
    if any(name in CRITICAL_FIELDS for name in conflicts):
        record["review_reason"] = f"conflicting values for {sorted(conflicts)}"
        quarantine(record, context)
        return record
    record["soft_conflicts"] = conflicts
    return record

Conflicts on critical fields — price, currency, identifier — should quarantine the record rather than picking a winner, because the cost of a wrong price downstream far exceeds the cost of a review. Conflicts on secondary fields are recorded and allowed through, which keeps the volume of quarantined records manageable.

structured_coverage is a per-record measure of how much came from machine-readable markup, and its average per host is the single best predictor of how exposed that host’s extraction is to a redesign.

Verification & Testing #

Handling a conflict between sourcesQuarantining every conflict makes the queue unmanageable; quarantine what matters.Handling a conflict between sourcesCritical fieldPrice, currency or identifierQuarantine the whole recordA human reviews the pairPattern usually fixes a whole hostSecondary fieldDescription, image, categoryHighest precedence winsConflict recorded on the recordCounted as a metric
Quarantining every conflict makes the queue unmanageable; quarantine what matters.
CASES = [
    ("json_ld_only.html",       {"price_minor": 129900, "source": "json-ld"}),
    ("microdata_only.html",     {"price_minor": 129900, "source": "microdata"}),
    ("both_agree.html",         {"price_minor": 129900, "source": "json-ld"}),
    ("both_disagree.html",      {"quarantined": True}),
    ("json_ld_partial.html",    {"availability_source": "microdata"}),  # per-field fallback
    ("formatting_differs.html", {"quarantined": False}),                # "1,299" vs "1299.00"
]

@pytest.mark.parametrize("fixture,expected", CASES, ids=[c[0] for c in CASES])
def test_precedence(fixture, expected):
    result = extract(load_fixture(fixture), site_config)
    for key, value in expected.items():
        assert result[key] == value

The json_ld_partial case is the one that proves the design: a page whose JSON-LD carries the price but not the availability should take the availability from microdata rather than reporting it missing. The formatting_differs case proves the normaliser is doing its job.

Watch two production metrics: conflict rate per field per host, and mean structured coverage per host. A rising conflict rate usually means one of the sources has gone stale — often a plugin that stopped updating — and falling coverage means a site has removed markup your extraction was quietly depending on.

Compliance & Operational Guardrails #

  • The source of every field is recorded on the record, so any value can be traced to its origin.
  • Conflicts on price, currency or identifier quarantine rather than resolve automatically.
  • The precedence order is configuration, not code, and a per-site override is recorded with a reason.
  • Personal data is classified identically regardless of which format supplied it.
  • Values are normalised before comparison so formatting differences do not register as conflicts.

Common Mistakes #

  1. Choosing a format wholesale. A partial block discards fields that another source had.
  2. Comparing raw strings for conflicts. Every record looks conflicted and the signal is abandoned.
  3. Preferring structured data for free text. Structured descriptions are frequently truncated summaries.

Frequently Asked Questions #

Why rank JSON-LD above microdata by default? #

It is more often maintained. JSON-LD is typically generated by a current plugin or template that is updated when the site changes, whereas microdata is embedded in themes that outlive several redesigns. This is a heuristic rather than a rule, so make it a per-site override — some platforms are the other way round, and the conflict metric will tell you which.

Should RDFa be implemented at all? #

Only after measuring. Add the extractor, log which fields it would have supplied that no other source did, and keep it if that number is meaningful for your targets. On many crawls it recovers a fraction of a percent and is not worth maintaining; on some verticals it is the primary format.

What if all sources are absent for a field? #

Record it as missing with source none and let the coverage metric carry the signal. A field consistently missing across a host is either genuinely absent from the pages or a selector that never matched, and the selector coverage metric distinguishes the two.

Should the precedence chain be the same for every field? #

No, and forcing it to be is the main reason teams abandon per-field resolution. Typed fields — price, currency, dates, identifiers — belong to the structured sources, where they are unambiguous. Free text belongs to the rendered page, which is what a reader sees. Images are usually best from structured data, because the block names the primary image explicitly while the markup requires guessing. Encoding those three groups as configuration rather than as one global order takes a few lines and removes most of the surprises.

How should conflicts be reviewed in practice? #

Sample rather than exhaust. A weekly review of ten conflicted records per host, chosen from those nearest the decision boundary, resolves the pattern for that host quickly — usually to a stale plugin, a tax-inclusive versus exclusive difference, or a personalised value. Fixing the pattern removes the whole group, whereas reviewing conflicts individually is unbounded work that never converges.

Should the resolution run before or after validation? #

Before. Resolution decides what the record’s values are; validation decides whether those values are acceptable. Running validation per source and then choosing between validated candidates does the work several times over and produces a confusing failure mode where a field is rejected from one source and accepted from another. Resolve first, then validate the single assembled record, and quarantine it as a whole if it fails.