Compliant Web Scraping

Chaining XPath Axes for Table Extraction #

Specification tables, definition lists and comparison grids share a property that defeats CSS entirely: the value you want has no identity of its own, and is located only by its relationship to a label. XPath axes express exactly that relationship, and chaining them handles the awkward cases — merged cells, multi-row headers, values split across nested elements. This guide covers the patterns, as part of XPath vs CSS selectors for scraping in the Data Parsing & Transformation Pipelines section.

Problem Framing #

A typical specification table looks like this:

Axes that earn their place in extractionancestor::x[1] is the nearest match, not the outermost.Axes that earn their place in extractionAxisReads asTypical usefollowing-siblingThe next cell alongLabel to valuepreceding-siblingThe cell beforeValue to labelancestorThe container aboveClimb to a carddescendantAnything insideFilter rows by marker
ancestor::x[1] is the nearest match, not the outermost.
<table class="specs">
  <tr><th>Weight</th><td>1.4 kg</td></tr>
  <tr><th>Dimensions</th><td>30 × 21 × 2 cm</td></tr>
  <tr><th>Material</th><td>Aluminium</td></tr>
</table>

There is nothing distinguishing 1.4 kg except that it sits beside a cell reading “Weight”. A CSS selector can reach the second cell of the first row — tr:first-child td — but that encodes row order, which changes whenever the site adds or reorders a specification. The value is located by its label, and only XPath can say so.

Real tables complicate this in four ways: cells spanning rows or columns, headers in a separate thead with a body that has no th at all, labels wrapped in nested markup so that text() returns nothing, and layouts using div elements with table-like styling rather than table elements at all.

Step-by-Step Implementation #

1. The label-to-value pattern #

Extracting a value by its labelnormalize-space on the element, not text(), survives a wrapped label.Extracting a value by its label1Match the labelwith normalize-space2Step to thesibling cell3Read full textcontent4Record the labelthat matched
normalize-space on the element, not text(), survives a wrapped label.
def value_for_label(tree, label: str) -> str | None:
    """The cell that follows a label cell, wherever the row sits in the table."""
    nodes = tree.xpath(
        "//tr[th[normalize-space(.)=$label] or td[normalize-space(.)=$label]]"
        "/td[not(normalize-space(.)=$label)][1]",
        label=label,
    )
    return " ".join(nodes[0].text_content().split()) if nodes else None

Three details do the work. Using normalize-space(.) rather than text() matches a label wrapped in a <span> or a <strong>, because . takes the element’s full text content while text() returns only its direct text children. Accepting either th or td for the label handles tables that use td throughout. And passing the label as an XPath variable rather than formatting it into the string is both faster — the expression stays compiled — and safe when the label contains a quotation mark.

2. Chain axes to climb and descend #

The axes that earn their place in extraction work are ancestor, following-sibling, preceding-sibling and descendant. Chaining them expresses “find the container of this thing, then a different thing inside it”.

# From a price badge, climb to the product card, then descend for the title.
titles = tree.xpath(
    "//span[@data-testid='price']/ancestor::article[1]//h3[1]/text()"
)

# From a section heading, take everything until the next heading of the same level.
section_rows = tree.xpath(
    "//h2[normalize-space()='Specifications']"
    "/following-sibling::*[not(self::h2)][count(preceding-sibling::h2[1]"
    "  [normalize-space()='Specifications'])=1]"
)

# Definition list: the value is a sibling of the term.
material = tree.xpath(
    "//dt[normalize-space()='Material']/following-sibling::dd[1]/text()"
)

ancestor::article[1] is the nearest matching ancestor, not the outermost — a distinction that matters on nested layouts and is the opposite of what many people expect from reading the expression left to right.

The section pattern is worth keeping: “everything between this heading and the next one of the same level” has no CSS equivalent and appears constantly in editorial and documentation markup.

3. Handle spans by expanding the grid #

Merged cells break positional reasoning entirely. Rather than compensating in the selector, expand the table into a rectangular grid once and index into it.

def expand_table(table) -> list[list[str]]:
    """Return a rectangular grid with rowspan and colspan expanded."""
    grid: list[list[str | None]] = []
    for row_index, row in enumerate(table.xpath(".//tr")):
        while len(grid) <= row_index:
            grid.append([])
        col = 0
        for cell in row.xpath("./th|./td"):
            while col < len(grid[row_index]) and grid[row_index][col] is not None:
                col += 1
            text = " ".join(cell.text_content().split())
            rowspan = int(cell.get("rowspan", 1) or 1)
            colspan = int(cell.get("colspan", 1) or 1)
            for r in range(row_index, row_index + rowspan):
                while len(grid) <= r:
                    grid.append([])
                while len(grid[r]) < col + colspan:
                    grid[r].append(None)
                for c in range(col, col + colspan):
                    grid[r][c] = text
            col += colspan
    width = max((len(r) for r in grid), default=0)
    return [[(cell or "") for cell in row] + [""] * (width - len(row)) for row in grid]

Expanding once and working on the grid is both simpler and more correct than trying to express span-awareness in a selector. It also makes the row-count assertion trivial, which is the check that catches a table whose extraction silently missed a section.

4. Cope with div-based pseudo-tables #

Layouts built from div elements with grid or flex styling carry no table semantics at all, so the label relationship has to be inferred from ARIA roles or from repeated structure.

def value_for_label_divs(tree, label: str) -> str | None:
    """Label/value pairs in a div layout, via ARIA roles or sibling structure."""
    nodes = tree.xpath(
        "//*[@role='row'][.//*[@role='rowheader'][normalize-space(.)=$label]]"
        "//*[@role='cell'][1]",
        label=label,
    )
    if nodes:
        return " ".join(nodes[0].text_content().split())

    nodes = tree.xpath(
        "//*[normalize-space(.)=$label][not(*)]/following-sibling::*[1]",
        label=label,
    )
    return " ".join(nodes[0].text_content().split()) if nodes else None

The [not(*)] predicate restricts the fallback to leaf elements — without it, an outer container whose entire text content happens to equal the label will match, and the “following sibling” is then some unrelated block. It is a small guard that removes a whole class of wrong answers.

Prefer the ARIA form where it exists: sites that add roles do so to make their layout accessible, and accessibility markup is maintained for the same reasons structured data is.

Verification & Testing #

Table extraction fixturesExpanding spans into a grid is simpler than expressing them in a selector.Table extraction fixtures✓Header cells and data cells in the same row✓A table using td throughout, with no th at all✓A label wrapped in a span or a strong element✓Merged cells spanning rows and columns✓A div grid with and without ARIA roles
Expanding spans into a grid is simpler than expressing them in a selector.
CASES = [
    ("simple_th_td.html",    "Weight",     "1.4 kg"),
    ("all_td.html",          "Weight",     "1.4 kg"),
    ("nested_label.html",    "Weight",     "1.4 kg"),   # label inside a <span>
    ("rowspan.html",         "Material",   "Aluminium"),
    ("div_grid_aria.html",   "Weight",     "1.4 kg"),
    ("div_grid_plain.html",  "Weight",     "1.4 kg"),
    ("label_absent.html",    "Weight",     None),
]

@pytest.mark.parametrize("fixture,label,expected", CASES, ids=[c[0] for c in CASES])
def test_label_lookup(fixture, label, expected):
    tree = html.fromstring(load_fixture(fixture))
    assert value_for_label(tree, label) or value_for_label_divs(tree, label) == expected

def test_grid_expansion_is_rectangular():
    grid = expand_table(html.fromstring(load_fixture("rowspan.html")).xpath("//table")[0])
    assert len({len(row) for row in grid}) == 1        # every row the same width

The label_absent case matters as much as the positive ones: a lookup for a label the page does not carry must return None rather than a value from somewhere else, and a chained expression with a loose predicate will happily return something.

Compliance & Operational Guardrails #

  • Extracted table rows carry the source URL and fetch time like any other record.
  • The row count is compared against the page’s own stated total where one exists.
  • Label lookups are recorded with the label matched, so a renamed label is visible as a coverage drop.
  • Expressions are compiled once and parameterised, never string-formatted from data.
  • Personal data appearing in table cells is classified exactly as elsewhere.

Common Mistakes #

  1. Using text() instead of normalize-space(.). A label wrapped in any inline element stops matching.
  2. Ignoring rowspan and colspan. Column alignment shifts silently and every value after the merge is wrong.
  3. Formatting labels into expressions. It defeats compilation and breaks on a quotation mark in the label.
  4. Assuming ancestor::x[1] is the outermost. It is the nearest, which is usually what you want and rarely what people expect.
  5. Extracting positionally when a label exists. Any reordering of rows silently changes what the extractor returns.

Frequently Asked Questions #

Is there a performance cost to chained axes? #

Modest, and dominated by the document parse. ancestor and preceding-sibling are more expensive than descendant traversal, but on a typical page the difference is microseconds. Compile the expression once and evaluate it relative to the smallest containing element rather than from the document root, and the cost disappears entirely — as the performance comparison shows in detail.

What if the same label appears twice on a page? #

Scope the lookup to a container first — the specification table, the section, the card — then apply the label expression within it. A document-wide lookup for a common label like “Price” on a page with related products will match the wrong one, and the multiplicity metric is what tells you it is happening.

How should labels be matched when wording varies? #

Keep a small synonym list per field in the site’s configuration and try each in order, recording which one matched. Sites rename “Weight” to “Item weight” without warning, and a configurable list makes that a data change rather than a code change. Avoid fuzzy matching — it eventually matches something it should not, and the resulting wrong value is harder to notice than a missing one.

How should a table with multi-row headers be handled? #

Expand the grid first, then reconstruct the header by concatenating the header cells above each column. A two-row header where the first row spans groups and the second names the columns produces composite labels such as 2026 / Revenue and 2026 / Margin, which are unambiguous and stable. Attempting to express that in a selector rather than in the grid is possible and reliably unreadable.

What if the same table appears twice on a page? #

Scope to the container before applying any label lookup — a section heading, a tab panel, an ARIA label. A document-wide label lookup on a page carrying both a specification table and a comparison table will match whichever comes first, and the resulting value looks entirely plausible. Recording the container that was matched, alongside the label, is what makes the wrong-container case visible in the extraction report rather than silent in the data.