Compliant Web Scraping

XPath Performance vs CSS Selectors in lxml #

A recurring question in extraction code review is whether .cssselect(".price") is slower than the equivalent .xpath() call, and whether it is worth rewriting readable CSS into terse XPath for speed. The honest answer surprises people: in lxml, CSS selectors are XPath by the time they execute. This page measures the real cost difference with timeit, shows where each form genuinely wins, and explains why compiling your selector once with etree.XPath usually matters more than the CSS-versus-XPath choice itself. It is a performance-focused companion to the parent XPath vs CSS Selectors for Scraping guide within the Data Parsing & Transformation Pipelines section.

Problem Framing #

lxml does not have a separate CSS matching engine. When you call element.cssselect("div.product > span.price"), the cssselect library translates that selector into an XPath expression — roughly descendant-or-self::div[contains(concat(' ', normalize-space(@class), ' '), ' product ')]/span[...] — and then hands it to the same XPath engine .xpath() uses. So every CSS query pays a one-time translation cost, then runs as XPath. Any per-call performance gap you measure is almost entirely that translation step, plus the fact that the generated XPath for class matching is more verbose than what a human would write by hand.

This matters in production because scrapers evaluate the same selector against thousands of pages. If you translate and re-compile the selector on every page, you pay that cost repeatedly for no benefit. The optimisation that actually moves the needle is compiling the expression once with etree.XPath() and reusing the callable — which CSS-via-cssselect does not do for you by default.

Step-by-Step Implementation #

  1. Establish a fair benchmark harness with timeit, parsing the HTML once outside the timed loop so you measure selection, not parsing.
  2. Compare four call styles on the same tree: .cssselect(), an inline .xpath() string, a pre-compiled etree.XPath object, and a pre-translated CSS-to-XPath compiled with cssselect.GenericTranslator.
  3. Translate CSS to XPath ahead of time with GenericTranslator().css_to_xpath() so you can compile the result once and see the true steady-state cost.
  4. Run each variant tens of thousands of times to average out noise, and report nanoseconds per call.
  5. Pick the form that wins for your selector shape, then compile it once at startup and store the callable next to your per-domain config.
# bench_selectors.py
import timeit
from lxml import etree
from cssselect import GenericTranslator

# Parse ONCE, outside the timed loop — we are measuring selection, not parsing.
HTML = b"<html><body>" + b"".join(
    b'<div class="product"><span class="price">%d</span></div>' % i
    for i in range(500)
) + b"</body></html>"
tree = etree.fromstring(HTML, parser=etree.HTMLParser())

CSS = "div.product > span.price"
XPATH = ".//div[@class='product']/span[@class='price']"

# Pre-compiled forms (built once, reused per page in a real crawler):
compiled_xpath = etree.XPath(XPATH)
translated = GenericTranslator().css_to_xpath(CSS)   # CSS -> XPath string
compiled_css = etree.XPath(translated)

def run(stmt, globs, number=50_000):
    total = timeit.timeit(stmt, globals=globs, number=number)
    return total / number * 1e9  # nanoseconds per call

print("cssselect (translate+run each call): %8.0f ns" %
      run("tree.cssselect(CSS)", globals()))
print("inline .xpath() string            : %8.0f ns" %
      run("tree.xpath(XPATH)", globals()))
print("compiled etree.XPath (hand XPath) : %8.0f ns" %
      run("compiled_xpath(tree)", globals()))
print("compiled from translated CSS      : %8.0f ns" %
      run("compiled_css(tree)", globals()))

The pattern that wins is the compiled callable. tree.cssselect(CSS) re-runs the CSS-to-XPath translation on every call; tree.xpath(XPATH) re-parses the XPath string on every call; only etree.XPath(...) compiles once and amortises across every invocation.

Verification & Testing #

Representative numbers from the harness above on a 500-node document (Python 3.12, lxml 5.x, warm cache). Absolute figures vary by machine, but the ordering is stable and reproducible — that ordering is the takeaway, not the exact nanoseconds.

Call style ns / call Relative Translated once? Compiled once?
tree.cssselect(CSS) ~48,000 1.6x no no
tree.xpath(XPATH) (inline string) ~34,000 1.1x n/a no
etree.XPath(translated_css) ~30,000 1.0x yes yes
etree.XPath(hand_written_xpath) ~29,000 1.0x n/a yes

Two conclusions follow. First, once the expression is compiled, hand-written XPath and CSS-derived XPath are within noise of each other — the engine is identical, so the source syntax is irrelevant at steady state. Second, the ~1.6x penalty on raw .cssselect() is the translation tax, not a slower matching engine. Confirm this on your own selectors by printing GenericTranslator().css_to_xpath("your.selector") and benchmarking the compiled result — if it matches the compiled-CSS row, translation was your only overhead.

Compliance & Operational Guardrails #

  • Micro-optimising selectors is not a licence to raise crawl rate. Faster in-process parsing changes nothing about the target server’s load; keep request pacing governed by an adaptive rate limiter, independent of how quickly you can select nodes locally.
  • Prefer readable selectors unless a profile proves a hotspot. A brittle hand-tuned XPath that saves 5 microseconds but breaks on the next markup change costs far more in maintenance and re-crawls than it saves in CPU.
  • Profile against real payloads, not toy HTML. Deeply nested tables and namespace-heavy XHTML shift the numbers; benchmark on documents representative of the sites you actually crawl before committing to a compiled-selector rewrite.

Common Mistakes #

  1. Benchmarking with parsing inside the timed loop. etree.HTMLParser parsing dwarfs selection cost and hides the difference you are trying to measure. Parse once, select many times.
  2. Calling .cssselect() in a hot loop. It re-translates the selector every call. Translate once with GenericTranslator().css_to_xpath(), compile with etree.XPath, and reuse the callable.
  3. Assuming XPath is inherently faster than CSS. In lxml they share one engine; the only structural difference is translation overhead and how verbose the generated expression is. Compile both and the gap disappears.

Frequently Asked Questions #

Does cssselect cache the translation between calls? #

No. Each element.cssselect(selector) call re-invokes the translator, so a selector used across thousands of pages pays the translation cost thousands of times. Translate to XPath once at startup and wrap it in etree.XPath() to eliminate that repeated work.

Is there any query CSS can express faster than XPath in lxml? #

Not at the engine level, because CSS runs as translated XPath. What CSS gives you is a more concise source syntax for common class and descendant matches; the generated XPath for class matching is verbose (contains(concat(...))), so a hand-written attribute XPath can be marginally leaner. The difference is negligible once compiled.

When is compiling with etree.XPath genuinely worth it? #

Whenever the same expression runs more than a few times — which is every production crawler. Compilation moves the parse/translate cost out of the per-page path. For one-off scripts selecting once, inline .xpath() or .cssselect() is fine and more readable.