Parsing Schema.org Product Data From JSON-LD #

Product markup is the most commercially useful structured data on the web and the most inconsistently implemented. The same schema supports a single item with one price, a product with fifty variants each with its own offer, and an aggregate range with no individual prices at all — and an extractor that handles only the first shape will silently mis-read the other two. This guide covers the shapes that occur in practice, as part of extracting JSON-LD and microdata in the Data Parsing & Transformation Pipelines section.

Problem Framing #

Three variations account for most extraction bugs.

None of these raise; they produce a plausible record with a missing price.

Offer shape. offers may be a single object, a list of objects, or an AggregateOffer carrying lowPrice and highPrice instead of price. Code written against one shape produces None or a type error on the others.

Variant structure. A product with sizes and colours may be one node with several offers, several sibling Product nodes, or a parent node with hasVariant children. Which one you get depends on the platform, and the correct record grain differs accordingly.

Reference indirection. A @graph block often expresses relationships by identifier — the product’s brand is {"@id": "#brand-1"} and the brand node appears elsewhere in the graph. An extractor that reads the value directly gets a dictionary containing only an identifier.

None of these produce an exception. They produce a record with a missing price, a wrong currency, or a brand of None, which passes validation if those fields are optional and quietly degrades the dataset.

Step-by-Step Implementation #

1. Resolve the graph before reading anything #

Reference resolution needs a depth guard: self-referential graphs exist.

def build_index(nodes: list[dict]) -> dict[str, dict]:
    """Map @id to node so references can be resolved."""
    return {node["@id"]: node for node in nodes if isinstance(node.get("@id"), str)}

def resolve(value, index: dict[str, dict], depth: int = 0):
    """Follow {"@id": ...} references, with a depth guard against cycles."""
    if depth > 4:
        return value
    if isinstance(value, dict):
        if set(value.keys()) == {"@id"} and value["@id"] in index:
            return resolve(index[value["@id"]], index, depth + 1)
        return {k: resolve(v, index, depth + 1) for k, v in value.items()}
    if isinstance(value, list):
        return [resolve(v, index, depth + 1) for v in value]
    return value

The depth guard matters: self-referential graphs are rare but they exist, and an unguarded resolver recurses until the stack runs out on a page that otherwise parses fine.

2. Normalise the offer shape #

from decimal import Decimal, InvalidOperation

AVAILABILITY = {
    "instock": "in_stock", "outofstock": "out_of_stock",
    "preorder": "preorder", "backorder": "backorder",
    "discontinued": "discontinued", "limitedavailability": "limited",
    "onlineonly": "in_stock", "instoreonly": "in_store_only",
    "soldout": "out_of_stock",
}

def parse_money(value) -> Decimal | None:
    if value is None:
        return None
    text = str(value).replace(",", "").strip()
    try:
        return Decimal(text)
    except InvalidOperation:
        return None

def normalise_offers(offers) -> list[dict]:
    """Return a flat list of offers regardless of the shape encountered."""
    if offers is None:
        return []
    if isinstance(offers, dict):
        offers = [offers]

    out = []
    for offer in offers:
        if not isinstance(offer, dict):
            continue
        types = offer.get("@type", "")
        types = types if isinstance(types, list) else [types]
        is_aggregate = any("aggregateoffer" in str(t).lower() for t in types)

        if is_aggregate:
            low, high = parse_money(offer.get("lowPrice")), parse_money(offer.get("highPrice"))
            out.append({
                "price_low": low, "price_high": high,
                "price": low,                          # a range's representative price
                "currency": offer.get("priceCurrency"),
                "availability": AVAILABILITY.get(
                    str(offer.get("availability", "")).rsplit("/", 1)[-1].lower()),
                "offer_count": offer.get("offerCount"),
                "is_range": True,
            })
            # An aggregate may still carry the individual offers underneath.
            out.extend(normalise_offers(offer.get("offers")))
            continue

        price = parse_money(offer.get("price"))
        spec = offer.get("priceSpecification")
        if price is None and isinstance(spec, dict):
            price = parse_money(spec.get("price"))
        out.append({
            "price": price,
            "price_low": price, "price_high": price,
            "currency": offer.get("priceCurrency")
                        or (spec or {}).get("priceCurrency"),
            "availability": AVAILABILITY.get(
                str(offer.get("availability", "")).rsplit("/", 1)[-1].lower()),
            "sku": offer.get("sku"),
            "is_range": False,
        })
    return out

priceSpecification is the fallback that recovers a meaningful share of records. Platforms that need to express tax-inclusive and tax-exclusive prices frequently move the value there and leave price absent, and an extractor that only reads price reports those products as unpriced.

The availability map normalises the schema.org URL suffix to your own vocabulary. Matching on the suffix rather than the full URL handles both https://schema.org/InStock and the bare InStock, both of which occur.

3. Decide the record grain #

One product node with five offers can become one record with a price range, or five records with individual prices. The right answer depends on what the data is for, and the important thing is to decide explicitly rather than to inherit whatever the markup happened to produce.

def product_records(node: dict, index: dict, source_url: str, grain: str = "variant") -> list[dict]:
    node = resolve(node, index)
    offers = normalise_offers(node.get("offers"))
    brand = node.get("brand")
    brand_name = brand.get("name") if isinstance(brand, dict) else brand

    base = {
        "source_url": source_url,
        "title": node.get("name"),
        "brand": brand_name,
        "sku": node.get("sku") or node.get("mpn"),
        "gtin": next((node[k] for k in ("gtin13", "gtin12", "gtin8", "gtin") if node.get(k)), None),
        "category": node.get("category"),
        "source_format": "json-ld",
    }

    if grain == "product" or not offers:
        prices = [o["price"] for o in offers if o["price"] is not None]
        return [{
            **base,
            "price_minor": int(min(prices) * 100) if prices else None,
            "price_high_minor": int(max(prices) * 100) if prices else None,
            "currency": next((o["currency"] for o in offers if o["currency"]), None),
            "offer_count": len(offers) or None,
            "availability": next((o["availability"] for o in offers if o["availability"]), None),
        }]

    return [{
        **base,
        "variant_sku": offer.get("sku"),
        "price_minor": int(offer["price"] * 100) if offer["price"] is not None else None,
        "currency": offer["currency"],
        "availability": offer["availability"],
    } for offer in offers]

Storing minor units as integers removes floating-point drift permanently, and it is the same convention the dates and currencies validation guide applies to values scraped from visible markup — so records from both paths are directly comparable.

4. Handle the multi-node page #

Product pages frequently carry several Product nodes: the item itself, related products, and recently-viewed items. Taking the first is wrong roughly as often as it is right.

def primary_product(nodes: list[dict], index: dict, canonical_url: str) -> dict | None:
    """Pick the node the page is actually about."""
    products = nodes_of_type(nodes, "Product")
    if not products:
        return None
    if len(products) == 1:
        return products[0]

    # Prefer a node whose url or @id matches the canonical URL.
    for node in products:
        for key in ("url", "@id", "mainEntityOfPage"):
            value = node.get(key)
            value = value.get("@id") if isinstance(value, dict) else value
            if isinstance(value, str) and value.rstrip("/") == canonical_url.rstrip("/"):
                return node
    # Otherwise the most completely described node is the page's subject.
    return max(products, key=lambda n: len(n.keys()))

Matching against the canonical URL is the reliable discriminator; the completeness heuristic is a reasonable last resort because related-product stubs carry a name and an image and little else.

Verification & Testing #

Assuming two decimal places turns a zero-decimal price into a hundredfold error.

CASES = [
    ("single_offer.json",     {"price_minor": 129900, "currency": "GBP", "count": 1}),
    ("offer_list.json",       {"price_minor": 99900,  "currency": "GBP", "count": 5}),
    ("aggregate_offer.json",  {"price_minor": 89900,  "currency": "USD", "count": 1}),
    ("price_spec_only.json",  {"price_minor": 45000,  "currency": "EUR", "count": 1}),
    ("graph_references.json", {"brand": "Example Brand",                 "count": 1}),
    ("multi_product.json",    {"title": "The Actual Product",            "count": 1}),
]

@pytest.mark.parametrize("fixture,expected", CASES, ids=[c[0] for c in CASES])
def test_product_extraction(fixture, expected):
    nodes = json_ld_nodes(load_fixture(fixture))
    node = primary_product(nodes, build_index(nodes), "https://example.com/p/1")
    records = product_records(node, build_index(nodes), "https://example.com/p/1")
    assert len(records) == expected["count"]
    for key, value in expected.items():
        if key != "count":
            assert records[0][key] == value

In production, monitor the share of extracted products with a null price per host. A step change is almost always a platform upgrade that moved the value — into priceSpecification, into an aggregate, or into a variant structure — and the null rate surfaces it within a day.

Compliance & Operational Guardrails #

Seller and reviewer names in product markup are personal data and are classified accordingly.
Aggregate ratings are attributed to the source, never presented as your own measurement.
The record grain is an explicit configuration choice, recorded per site.
Prices are stored as integer minor units with an explicit currency code.
The raw structured block is retained briefly so a schema addition can be backfilled without a recrawl.

Common Mistakes #

Assuming offers is a dict. Lists and aggregates are equally common and produce silent nulls.
Reading only price. priceSpecification and lowPrice carry the value on a meaningful share of sites.
Taking the first Product node. Related-item stubs frequently appear before the page’s actual subject.

Frequently Asked Questions #

What if the structured price disagrees with the displayed price? #

Record both and flag the record. The usual causes are a cached block, a tax-inclusive versus tax-exclusive difference, or a personalised price shown to a session. None can be resolved by preferring one automatically, and a flagged sample reviewed weekly resolves the pattern for that host quickly.

Should variants become separate records? #

If prices or availability differ per variant and your use cares about that, yes. If you are tracking a headline price, a single record with a range is simpler and less duplicative. Whichever you choose, make it a per-site setting rather than an emergent property of the markup, so the grain of your dataset is something you can state.

How should missing currency be handled? #

Quarantine rather than assume. Currency can sometimes be inferred from the site’s locale or from a sibling offer, and where that inference is reliable it belongs in configuration as an explicit per-site default. Guessing from a symbol is the case to avoid entirely — several currencies share $ and kr.

Extracting JSON-LD and Microdata — parsing the graph this guide reads from.
Falling Back From JSON-LD to Microdata — what to do when the block is absent or incomplete.
Validating Scraped Dates and Currencies With Pydantic — the typing conventions these records feed into.