Extracting JSON-LD and Microdata #
The most reliable data on a page is usually the data the site publishes for machines. Structured markup — JSON-LD, microdata, RDFa — is authored deliberately, changes far less often than the visual layout, and carries typed values that a CSS selector can only approximate. A pipeline that reads structured data first and falls back to selectors second is markedly more stable than one built the other way round. This topic, part of the Data Parsing & Transformation Pipelines section, covers how to extract each format, reconcile them, and validate the result.
Core Principles: Why Structured Data Is the Better Anchor #
Three properties make structured markup the right first choice.
It survives redesigns. A site can change every class name, restructure its containers and adopt a new framework while leaving the JSON-LD block untouched, because that block exists for search engines rather than for readers. Selectors anchored to visual structure break; a Product schema does not.
It is typed. A price in the visible markup is the string £1,299.00 embedded in a span, and recovering the number and the currency requires parsing and inference. In JSON-LD it is {"price": "1299.00", "priceCurrency": "GBP"} — already separated, already unambiguous.
It is unambiguous about relationships. Which price belongs to which variant, which review belongs to which product, which image is primary — all questions requiring positional inference in the rendered markup and answered explicitly by the structured graph.
The trade-off is coverage: not every page carries it, some carry incomplete blocks, and some carry blocks that disagree with what the page displays. A production extractor therefore treats structured data as the preferred source with a defined fallback, rather than as the only one. That layering is the same resilient selector chain principle applied one level higher.
The Three Formats #
JSON-LD lives in a <script type="application/ld+json"> element, entirely separate from the visual markup. It is the easiest to parse and by far the most common on modern sites.
Microdata annotates the visible markup with itemscope, itemtype and itemprop attributes. It requires a tree walk, and its values come from element content or from specific attributes depending on the element type.
RDFa uses vocab, typeof and property attributes with similar semantics and different attribute names. It is the least common of the three on commercial sites, and a parser that handles microdata can usually be extended to cover it cheaply.
A page may carry more than one, and they may disagree. Extraction therefore needs a defined precedence, which is worked through in falling back from JSON-LD to microdata.
Technical Implementation #
Parsing the JSON-LD graph #
The naive implementation reads the first script block and takes its top-level object. Real pages are messier: several blocks, arrays at the top level, @graph wrappers, and nested entities referenced by identifier.
import json
from lxml import html
def json_ld_nodes(document_html: str) -> list[dict]:
"""Every JSON-LD entity on the page, flattened out of arrays and @graph wrappers."""
tree = html.fromstring(document_html)
nodes: list[dict] = []
for script in tree.xpath('//script[@type="application/ld+json"]'):
raw = (script.text_content() or "").strip()
if not raw:
continue
try:
payload = json.loads(raw)
except json.JSONDecodeError:
# Trailing commas and unescaped newlines are common; skip rather than crash.
continue
stack = payload if isinstance(payload, list) else [payload]
while stack:
item = stack.pop()
if not isinstance(item, dict):
continue
graph = item.get("@graph")
if isinstance(graph, list):
stack.extend(graph)
continue
nodes.append(item)
return nodes
def nodes_of_type(nodes: list[dict], wanted: str) -> list[dict]:
"""Match a schema type, tolerating both a string and a list in @type."""
out = []
for node in nodes:
types = node.get("@type", "")
types = types if isinstance(types, list) else [types]
if any(str(t).rsplit("/", 1)[-1].lower() == wanted.lower() for t in types):
out.append(node)
return outTwo robustness details matter. @type is frequently a list — a node can be both a Product and a Vehicle — and code that compares it to a string misses those. And types are sometimes fully qualified URLs, so comparing the final path segment rather than the whole value catches both spellings.
Walking microdata #
ITEMPROP_ATTR = {
"meta": "content", "audio": "src", "embed": "src", "iframe": "src",
"img": "src", "source": "src", "track": "src", "video": "src",
"a": "href", "area": "href", "link": "href",
"object": "data", "data": "value", "meter": "value", "time": "datetime",
}
def microdata_items(tree) -> list[dict]:
"""Extract every top-level itemscope as a nested dict."""
def value_of(node):
if node.get("itemscope") is not None:
return read_item(node)
attr = ITEMPROP_ATTR.get(node.tag)
if attr and node.get(attr):
return node.get(attr).strip()
return (node.text_content() or "").strip()
def read_item(scope) -> dict:
item: dict = {}
if scope.get("itemtype"):
item["@type"] = scope.get("itemtype").rsplit("/", 1)[-1]
for node in scope.iter():
if node is scope or not node.get("itemprop"):
continue
# Skip properties belonging to a nested scope; read_item handles those.
parent_scope = next((a for a in node.iterancestors()
if a.get("itemscope") is not None), None)
if parent_scope is not scope:
continue
for name in node.get("itemprop").split():
value = value_of(node)
if name in item:
existing = item[name]
item[name] = existing + [value] if isinstance(existing, list) else [existing, value]
else:
item[name] = value
return item
tops = [n for n in tree.iter()
if n.get("itemscope") is not None
and not any(a.get("itemscope") is not None for a in n.iterancestors())]
return [read_item(node) for node in tops]The attribute table is what makes microdata correct rather than approximately correct: a <meta itemprop="price" content="1299.00"> has no text content at all, and a walker reading text_content() for every element silently produces an empty price. Likewise <time itemprop="datePublished" datetime="2026-06-14"> carries the machine-readable value in an attribute while displaying something else entirely.
Normalising into your own schema #
Structured data is a source, not a destination. Map it into the same record shape your selectors produce, so downstream code never needs to know which path a value came from.
from decimal import Decimal
def product_from_json_ld(node: dict) -> dict:
offers = node.get("offers") or {}
if isinstance(offers, list):
offers = offers[0] if offers else {}
price = offers.get("price") or offers.get("lowPrice")
currency = offers.get("priceCurrency")
return {
"title": node.get("name"),
"sku": node.get("sku") or node.get("mpn"),
"brand": (node.get("brand") or {}).get("name") if isinstance(node.get("brand"), dict)
else node.get("brand"),
"price_minor": int(Decimal(str(price)) * 100) if price is not None else None,
"currency": currency,
"availability": str(offers.get("availability", "")).rsplit("/", 1)[-1] or None,
"source_format": "json-ld",
}Recording source_format on every record is worth the column. It lets you measure how much of the dataset came from structured markup versus from selectors, and a host whose structured coverage drops is worth investigating before the selector fallback starts carrying everything.
Compliance Boundaries #
Structured data is published for machines, which makes it the most clearly consented-to data on a page — but it is not exempt from anything.
-
isAccessibleForFree: false
The second item is easily overlooked. A publisher that marks a document as not freely accessible has stated the position in the same block your extractor is already reading, which makes honouring it nearly free — and ignoring it correspondingly harder to explain. The handling is set out in respecting paywalls and authentication boundaries.
Validating Structured Data Before Trusting It #
Structured markup is authored by people and generated by plugins, which means it is wrong often enough to need checking. Three classes of error recur, and each has a cheap detector.
Stale blocks. A cached or templated block carrying last month’s price while the page displays this month’s. Detected by comparing the structured value against the rendered one and recording the disagreement.
Placeholder values. Template defaults that were never filled in — a price of 0.00, a name of Product Name, a currency of XXX. Detected by a small deny-list per field, maintained from observation.
Structural nonsense. A Product whose offers is a string, a datePublished in the future, an AggregateRating with a ratingValue outside its own declared scale. Detected by validating against the schema’s own constraints.
from datetime import datetime, timezone
from decimal import Decimal
PLACEHOLDERS = {
"name": {"product name", "item title", "untitled", "n/a", "-"},
"currency": {"xxx", "cur", ""},
"brand": {"brand", "manufacturer", "n/a"},
}
def structured_warnings(record: dict) -> list[str]:
warnings = []
for field, bad in PLACEHOLDERS.items():
value = str(record.get(field, "")).strip().lower()
if value and value in bad:
warnings.append(f"{field}: placeholder value {value!r}")
price = record.get("price_minor")
if price is not None and price <= 0:
warnings.append("price: non-positive value from structured data")
published = record.get("date_published")
if published:
when = datetime.fromisoformat(published)
if when > datetime.now(timezone.utc):
warnings.append("date_published: in the future")
rating, best = record.get("rating"), record.get("rating_best") or 5
if rating is not None and not (0 <= Decimal(str(rating)) <= Decimal(str(best))):
warnings.append(f"rating: {rating} outside 0..{best}")
return warningsAttach the warnings to the record rather than discarding it. A structured block with a placeholder name and a correct price is still the best source for the price, and dropping the whole record because one field is templated loses more than it protects. Where a critical field is affected — price, currency, identifier — fall back to the next source in the chain and record that the fallback was triggered by a warning rather than by absence.
Track the warning rate per host as a metric. A host whose placeholder rate rises sharply has usually deployed a template change, and the structured extraction for that host needs a look before its output silently degrades.
Coverage, and Knowing How Exposed You Are #
The single most useful measure of an extraction’s fragility is how much of it comes from structured data. A record assembled entirely from a JSON-LD block will survive a visual redesign untouched; one assembled entirely from class-name selectors will not survive the next design-system release.
STRUCTURED_SOURCES = {"json-ld", "microdata", "rdfa"}
def coverage(field_sources: dict[str, str]) -> float:
if not field_sources:
return 0.0
return sum(1 for s in field_sources.values() if s in STRUCTURED_SOURCES) / len(field_sources)Exporting the mean coverage per host turns “which sites will break next?” from a guess into a ranked list. A host at 0.9 is safe; one at 0.1 is one redesign from an incident, and that is where fixture refreshes and selector review effort should go.
Coverage also has a diagnostic use. A sudden drop for one host almost always means the structured block stopped being emitted — a plugin disabled, a migration half-finished — and that is worth reporting to the site operator, since it degrades their search presence at the same time as your extraction.
Keeping the Raw Block #
Retain the structured block itself, briefly, alongside the raw response. It is small — usually a few kilobytes — and it makes a whole class of change cheap: adding a field to your schema becomes a reprocessing job over retained blocks rather than a full recrawl of every page.
The retention window should match the landing zone’s, and the same obligations apply: a Person, author or seller node in a structured block is personal data exactly as it would be in the visible markup, and it is subject to the same classification, minimisation and expiry rules described in GDPR compliance for scraped personal data. Structured data being machine-readable makes it easier to process, not less regulated.
Common Mistakes #
- Reading only the first script block. Pages routinely carry several — an organisation, a breadcrumb list, and the product — and the one you want is rarely first.
- Assuming
@typeis a string. Multi-typed nodes are common and a string comparison silently skips them. - Reading
text_content()for every microdata property.meta,time,linkandimgcarry their values in attributes. - Trusting structured data that contradicts the page. Stale or templated blocks exist; where the two disagree, record the conflict rather than silently preferring one.
- Discarding the structured block after extraction. Keeping it briefly makes a later schema addition a reprocessing job rather than a recrawl.
Frequently Asked Questions #
What if the JSON-LD is invalid JSON? #
Skip that block and fall back, but count it. Trailing commas, unescaped control characters and HTML entities inside string values are the usual causes, and a host with a consistently broken block is worth reporting to its operator — it is breaking their search presence too, so the report is usually welcome. Do not attempt heuristic repair; a mis-repaired block produces plausible wrong values rather than an obvious failure.
Should structured data always win over the visible content? #
For typed fields — price, currency, dates, identifiers — yes, because the structured form is unambiguous where the rendered form needs inference. For free text such as descriptions, prefer the visible content, which is what a reader actually sees; structured descriptions are frequently truncated or templated. Where the two disagree on a typed field, record both and flag the record for review.
How much coverage should I expect? #
On commercial sites — retail, recruitment, property, events — structured data coverage is high, often above 80% of detail pages. On editorial and niche sites it is much lower. Measure it per host as a metric rather than assuming, because it also tells you how exposed that host’s extraction is to a redesign.
Does microdata still matter given JSON-LD’s dominance? #
Yes, for two reasons: older platforms and long-lived templates still emit it, and some sites publish partial JSON-LD alongside complete microdata. A fallback chain that tries JSON-LD, then microdata, then RDFa, then selectors typically recovers several percent of records that a JSON-LD-only extractor drops.
How do I test a structured-data extractor without a live site? #
Save the raw structured blocks as fixtures rather than whole pages. They are small, they contain the entire input the extractor consumes, and they make the test suite fast enough to run on every commit. Cover the shapes that differ between platforms — a single offer, a list, an aggregate, a @graph with references, a multi-typed node, a page with several Product entries — and add a fixture whenever a real site surprises you. A corpus grown from production observation covers far more of the space than one written from the specification, because the specification does not describe the ways implementations get it wrong.
Should structured data be re-extracted when the schema changes? #
Yes, and that is precisely why the raw block is worth retaining briefly. Adding a field to your record — a GTIN, a shipping detail, an availability date — becomes a reprocessing job over retained blocks rather than a full recrawl, which is both faster and considerably kinder to the sites involved. Beyond the retention window a recrawl is unavoidable, which is a good argument for keeping the window long enough to cover a normal schema-change cycle.
Is structured data ever worth writing back to your own site? #
If you publish anything derived from a crawl, yes — for exactly the reasons this topic gives for reading it. Machine-readable output is stable across your own redesigns, unambiguous about types and relationships, and vastly cheaper for a consumer to parse than your rendered markup. The reciprocity is also worth noting: a pipeline that depends on other people’s structured data and publishes none of its own is taking advantage of a convention it does not contribute to.
Related guides #
- Parsing Schema.org Product Data From JSON-LD — the full product mapping with offers, variants and availability.
- Falling Back From JSON-LD to Microdata — precedence rules when a page carries more than one format.
- Writing Resilient Selectors That Survive Redesigns — the final fallback when no structured data exists.
- Schema Validation With Pydantic — typing the record once both sources have been reconciled.