Compliant Web Scraping

BeautifulSoup vs lxml for Large HTML Documents #

When a sitemap-driven crawl starts pulling 20 MB catalogue pages or a single archive dump that is hundreds of megabytes, the parser you chose for convenience becomes the thing that kills the worker. BeautifulSoup builds a rich, mutable Python object for every tag, which is wonderful ergonomically and expensive in memory; lxml holds a compact C tree and, critically, can stream a document it never fully materialises. This page compares the two on memory and throughput at scale, and shows the lxml.etree.iterparse target-parser pattern that lets you process documents larger than RAM. It extends the parent XPath vs CSS Selectors for Scraping guide within the Data Parsing & Transformation Pipelines section.

Problem Framing #

The two libraries fail at scale for different reasons. BeautifulSoup (on its default html.parser, or even wrapping lxml) constructs a full tree of Tag, NavigableString, and Comment objects — each a Python object with its own dictionary and reference overhead. A document with a few hundred thousand nodes can inflate to five to ten times its byte size in resident memory, and that entire tree must exist before you touch the first element. For a page or two this is invisible; run it across a fleet of concurrent workers on large pages and you exhaust the container’s memory limit and start getting OOM-killed.

lxml in DOM mode (etree.HTML) is far leaner because the tree lives in C, but it still loads the whole document. The real escape hatch is streaming: iterparse emits elements as they are parsed and lets you delete each subtree once consumed, so peak memory tracks the largest single record rather than the whole file. The tradeoff is that streaming code is more constrained to write than the “load it all, query freely” model.

Step-by-Step Implementation #

  1. Reach for the DOM only when the document fits comfortably in memory and you need arbitrary back-and-forth navigation. Use lxml’s etree.HTML for speed, or BeautifulSoup when markup is badly broken and you value forgiving traversal over throughput.
  2. Switch to streaming above a size threshold (a practical cutoff is anything over a few MB, or any unbounded feed). Use lxml.etree.iterparse with a tag filter so it only surfaces the elements you care about.
  3. Free each subtree after processing. Call element.clear() and delete preceding siblings so the parser does not accumulate the whole tree — this is what keeps memory flat.
  4. Use html.parser=True on iterparse for real-world (non-XML-clean) HTML, and target the repeating record element (tr, article, item) as your unit of work.
  5. Hand off convenience-heavy cleanup to BeautifulSoup only for the small fragments where its API earns its cost, not the whole document.
# stream_large_html.py
from lxml import etree

def stream_rows(path: str, row_tag: str = "tr"):
    """Yield dicts from a huge HTML table without loading the whole file.

    Peak memory tracks the largest single row, not the document size.
    """
    context = etree.iterparse(
        path,
        events=("end",),
        tag=row_tag,
        html=True,          # tolerate real-world, non-well-formed HTML
        recover=True,       # do not abort on unclosed tags
    )
    for _event, element in context:
        cells = [ "".join(td.itertext()).strip()
                  for td in element.iterfind(".//td") ]
        if cells:
            yield cells
        # Release this subtree AND its already-seen previous siblings.
        element.clear()
        while element.getprevious() is not None:
            del element.getparent()[0]
    del context


if __name__ == "__main__":
    seen = 0
    for row in stream_rows("catalogue_500mb.html", row_tag="tr"):
        seen += 1  # push `row` to your validation/storage stage here
    print(f"streamed {seen} rows at flat memory")

The element.clear() plus previous-sibling deletion is the load-bearing part. Without it, iterparse still retains every element it has emitted and you lose the memory advantage entirely — a common and silent mistake.

Verification & Testing #

Measure resident memory, not wall-clock alone; the whole point at scale is the memory ceiling. tracemalloc gives a reproducible in-process peak without external tooling.

# measure_peak.py
import tracemalloc
from bs4 import BeautifulSoup
from lxml import etree
from stream_large_html import stream_rows

html = open("catalogue_50mb.html", "rb").read()

tracemalloc.start()
soup = BeautifulSoup(html, "html.parser")
rows_bs = soup.find_all("tr")
_, peak_bs = tracemalloc.get_traced_memory()
tracemalloc.stop()

tracemalloc.start()
count = sum(1 for _ in stream_rows("catalogue_50mb.html"))
_, peak_stream = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"BeautifulSoup DOM peak : {peak_bs / 1e6:8.1f} MB")
print(f"lxml iterparse peak    : {peak_stream / 1e6:8.1f} MB")

Indicative results on a 50 MB table with roughly 300k rows (Python 3.12; figures scale with document size, so treat the ratios as the signal):

Approach Peak memory Full doc in RAM? Streams? Handles broken HTML
BeautifulSoup (html.parser) ~520 MB yes no excellent
BeautifulSoup(html, "lxml") ~410 MB yes no very good
lxml DOM (etree.HTML) ~180 MB yes no good (recover=True)
lxml.etree.iterparse (cleared) ~12 MB no yes good (html=True)

The streaming row shows the effect clearly: peak memory is roughly constant regardless of file size because only one record lives at a time. The BeautifulSoup rows scale linearly with document size and hit the memory wall first. To prove your own clearing logic works, run stream_rows on a file several times larger than the box’s RAM allowance — if it completes, memory is genuinely flat.

Compliance & Operational Guardrails #

  • Set a hard document-size cap before parsing. A hostile or misconfigured server can return an unbounded response that OOM-kills the worker; cap the download and route oversized bodies to streaming, treating parser memory as a resource budget the same way you treat request rate.
  • Streaming does not exempt you from politeness. Processing a huge file efficiently locally says nothing about how you fetched it; keep crawl pacing under Exponential Backoff and Retry Logic and honour robots.txt regardless of parser.
  • Minimise what you retain. iterparse makes it easy to extract only the fields you need and drop the rest before anything is stored, which supports data-minimisation obligations far better than holding a full DOM you then filter.

Common Mistakes #

  1. Forgetting to clear() and delete previous siblings in iterparse. The parser keeps everything it has emitted, so memory grows exactly as if you loaded the whole tree — the streaming benefit silently evaporates.
  2. Using BeautifulSoup with the pure-Python html.parser at scale. It is the slowest and heaviest combination. If you need BeautifulSoup’s API, at least pass "lxml" as the backend; if you need scale, stream instead.
  3. Streaming when you actually need random access. iterparse is forward-only; code that must jump to ancestors or arbitrary siblings fights the model. For complex within-record navigation, buffer the single record subtree and query it as a small DOM before clearing.

Frequently Asked Questions #

If BeautifulSoup can use lxml as its backend, why is it still heavier? #

Because BeautifulSoup builds its own layer of Python Tag and NavigableString objects on top of whatever backend parses the bytes. The lxml backend speeds up the initial parse, but the resulting BeautifulSoup tree is still a full graph of Python objects, so peak memory stays far above raw lxml. You pay for the friendly API in resident memory.

Can I still use CSS selectors or XPath while streaming? #

Within a single cleared subtree, yes — treat each emitted element as a tiny DOM and run .xpath() or .cssselect() against it before clearing. You cannot run a document-wide selector across a streamed file, because the whole document never exists at once. For selector-level performance detail, see the XPath performance vs CSS selectors in lxml comparison.

When is BeautifulSoup’s convenience worth the memory cost? #

On small-to-medium pages where the markup is badly malformed and you need forgiving traversal, sibling-walking, or in-place mutation. Its resilience to broken HTML and readable API genuinely save development time there. The cost only becomes decisive when documents are large or workers are many and concurrent.