Compliant Web Scraping

Detecting Character Encoding of Scraped HTML #

Bytes are not text until something decides which encoding they are in, and every layer of the stack offers a different opinion. Getting the decision wrong produces a dataset speckled with replacement characters and mojibake in exactly the fields that matter — names, addresses, currency symbols, anything outside ASCII. This guide implements a resolution order that is correct and, more importantly, observable, as part of advanced HTML parsing with BeautifulSoup in the Data Parsing & Transformation Pipelines section.

Problem Framing #

Four sources can declare an encoding and they frequently disagree: a byte-order mark at the start of the body, the charset parameter of the Content-Type header, a <meta charset> declaration inside the document, and the HTTP client’s own default when nothing else is present.

Encoding resolution orderRecord which source won: a wrong declaration is only visible if you logged it.Encoding resolution orderByte-order markunambiguous when present, checked firstContent-Type charsetset by the server or a CDNMeta charset in the headwritten by a template, often olderStatistical detectiona last resort, unreliable on short textDeclared fallbackUTF-8, with the failure recorded
Record which source won: a wrong declaration is only visible if you logged it.

Disagreement is common because the header is set by the server or CDN while the meta tag is written by a template that may be years older. A site that migrated to UTF-8 and left charset=iso-8859-1 in a header is entirely ordinary.

The failure is quiet. Decoding UTF-8 bytes as Latin-1 never raises — every byte is a valid Latin-1 character — it simply produces é where é belonged. Nothing errors, validation passes, and the corruption is discovered when a user searches for a name and finds nothing.

Step-by-Step Implementation #

1. Resolve in a defined order, and record which source won #

Strict decode first, or corruption becomes invisibleThe point of a strict attempt is the failure it produces.Strict decode first, or corruption becomes invisibleReplace from the startNothing ever raisesNothing is ever loggedNames arrive with replacement marksFound by a user, months laterStrict, then fall backA wrong declaration surfacesThe fallback is recordedReplacement counts become a metricFound on the day it starts
The point of a strict attempt is the failure it produces.
import re
from dataclasses import dataclass

META_CHARSET = re.compile(rb'charset\s*=\s*["\']?\s*([\w\-]+)', re.I)
BOMS = (
    (b"\xef\xbb\xbf", "utf-8-sig"),
    (b"\xff\xfe\x00\x00", "utf-32-le"), (b"\x00\x00\xfe\xff", "utf-32-be"),
    (b"\xff\xfe", "utf-16-le"), (b"\xfe\xff", "utf-16-be"),
)

@dataclass(frozen=True)
class Decoded:
    text: str
    encoding: str
    source: str          # "bom" | "http-header" | "meta-tag" | "detector" | "fallback"
    replacements: int    # count of U+FFFD, if a lossy decode was needed

def resolve_encoding(body: bytes, content_type: str | None) -> tuple[str, str]:
    for bom, encoding in BOMS:
        if body.startswith(bom):
            return encoding, "bom"
    if content_type and "charset=" in content_type.lower():
        match = META_CHARSET.search(content_type.encode("latin-1", "ignore"))
        if match:
            return match.group(1).decode("ascii", "ignore").lower(), "http-header"
    # The declaration must appear in the first 1024 bytes to be honoured.
    match = META_CHARSET.search(body[:1024])
    if match:
        return match.group(1).decode("ascii", "ignore").lower(), "meta-tag"
    return "", "none"

Restricting the meta search to the first kilobyte matches how browsers behave and avoids matching a charset mentioned in body text or in a script — a real source of wrong answers on pages that discuss encodings.

2. Decode strictly, then fall back deliberately #

import codecs

ALIASES = {
    "ascii": "utf-8",            # ASCII is a UTF-8 subset; UTF-8 is the safer superset
    "latin1": "cp1252",          # servers say latin-1 and mean windows-1252
    "iso-8859-1": "cp1252",      # the HTML spec makes this substitution too
    "utf8": "utf-8", "utf_8": "utf-8",
    "gb2312": "gb18030",         # superset, decodes more real-world content
    "shift-jis": "shift_jis",
}

def normalise_encoding(name: str) -> str | None:
    candidate = ALIASES.get(name.strip().lower(), name.strip().lower())
    try:
        codecs.lookup(candidate)
    except LookupError:
        return None
    return candidate

def decode_body(body: bytes, content_type: str | None,
                site_default: str | None = None) -> Decoded:
    declared, source = resolve_encoding(body, content_type)
    encoding = normalise_encoding(declared) if declared else None

    if encoding:
        try:
            return Decoded(body.decode(encoding, errors="strict"), encoding, source, 0)
        except UnicodeDecodeError:
            pass                                   # declaration was wrong; keep looking

    for candidate, label in ((site_default, "site-default"), ("utf-8", "fallback")):
        if not candidate:
            continue
        try:
            return Decoded(body.decode(candidate, errors="strict"), candidate, label, 0)
        except (UnicodeDecodeError, LookupError):
            continue

    # Nothing decoded cleanly: accept a lossy decode but count the damage.
    text = body.decode(encoding or "utf-8", errors="replace")
    return Decoded(text, encoding or "utf-8", f"{source}-lossy", text.count("�"))

The Latin-1 to Windows-1252 substitution is not a hack — it is what the HTML specification requires, because servers declare iso-8859-1 while serving bytes containing smart quotes and dashes that only exist in Windows-1252. Honouring the declaration literally produces control characters where punctuation belonged.

Attempting a strict decode first is what makes the failure observable. A pipeline that decodes with errors="replace" from the outset never learns that a declaration was wrong.

3. Use a detector only as a last resort, and record that you did #

def detect_encoding(body: bytes) -> tuple[str | None, float]:
    """Statistical detection. Slow and fallible — a last resort, never a first choice."""
    try:
        from charset_normalizer import from_bytes
    except ImportError:
        return None, 0.0
    best = from_bytes(body[:65536]).best()
    if best is None:
        return None, 0.0
    return str(best.encoding), 1.0 - min(best.chaos, 1.0)

Detection is genuinely useful when nothing is declared, and genuinely unreliable on short documents or ones that are mostly ASCII with a handful of non-ASCII characters — which describes a large share of pages. Sample generously, treat a low confidence as no answer, and record source="detector" so the resulting text can be re-examined if it turns out wrong.

4. Instrument it #

def record_decode(decoded: Decoded, host: str) -> None:
    metrics.decode_source.labels(host=host, source=decoded.source).inc()
    if decoded.replacements:
        metrics.decode_replacements.labels(host=host).inc(decoded.replacements)
        if decoded.replacements > 20:
            quarantine_encoding(host=host, encoding=decoded.encoding,
                                source=decoded.source, count=decoded.replacements)

The replacement count is the metric that matters. A handful of replacement characters across a large crawl is normal — genuinely malformed bytes exist. A step change on one host means its encoding changed or its declaration became wrong, and catching that on the day it happens avoids weeks of corrupted records.

Verification & Testing #

Declared encodings worth substitutingHonouring iso-8859-1 literally turns smart quotes into control characters.Declared encodings worth substitutingDeclaredDecode asReasoniso-8859-1windows-1252Spec-mandated substitutionlatin1windows-1252Servers mean 1252asciiutf-8Safer supersetgb2312gb18030Decodes more real content
Honouring iso-8859-1 literally turns smart quotes into control characters.
CASES = [
    (b"\xef\xbb\xbfcaf\xc3\xa9", None, "café", "bom"),
    (b"caf\xc3\xa9", "text/html; charset=utf-8", "café", "http-header"),
    (b'<meta charset="utf-8">caf\xc3\xa9', None, "café", "meta-tag"),
    # Header says latin-1, bytes are Windows-1252: the smart quote must survive.
    (b"\x93quoted\x94", "text/html; charset=iso-8859-1", "“quoted”", "http-header"),
    # Header lies (says latin-1, bytes are UTF-8): strict decode succeeds as latin-1,
    # so this case documents why observability matters more than cleverness.
    (b"caf\xc3\xa9", "text/html; charset=utf-8", "café", "http-header"),
]

@pytest.mark.parametrize("body,ctype,expected,source", CASES)
def test_decoding(body, ctype, expected, source):
    result = decode_body(body, ctype)
    assert expected in result.text
    assert result.source == source

def test_lossy_decode_is_counted():
    result = decode_body(b"\xff\xfe\xff invalid", "text/html; charset=utf-8")
    assert result.replacements > 0

The Windows-1252 case is the one to keep permanently: it is the single most common real-world encoding bug, and a naive implementation turns typographic quotes into control characters that then propagate into every downstream field.

For a stronger check, add a sentinel assertion on a per-site fixture containing known non-ASCII text, and assert the exact expected string. It catches double-encoding, which produces text that is technically valid and visibly wrong.

Compliance & Operational Guardrails #

  • Decoding happens once, at the transport boundary; nothing downstream re-decodes.
  • The encoding and the source of the decision are recorded on the record.
  • Replacement-character counts are exported per host and alerted on.
  • A site-specific encoding override is configuration with a recorded reason, never a code change.
  • Raw bytes are retained briefly in the landing zone so a mis-decode can be re-run without a recrawl.

Common Mistakes #

  1. Decoding with errors="replace" from the start. Corruption becomes invisible and permanent.
  2. Honouring iso-8859-1 literally. Punctuation becomes control characters on a large share of European sites.
  3. Re-encoding to “fix” text. Each round trip compounds the damage and double-encoded artefacts are irreversible.
  4. Trusting the detector on short documents. Confidence is low and the answer is frequently wrong.
  5. Decoding in the parser rather than at the boundary. Different parsers make different assumptions and the record’s text depends on which one ran.

Frequently Asked Questions #

What if the header and the meta tag disagree? #

The header wins by specification, and that is the right default — but record the disagreement. A host where the two consistently differ is worth a site-specific override, particularly if the header comes from a CDN that nobody has reconfigured since a migration.

Should the raw bytes be kept? #

Briefly, in the landing zone. A mis-decode discovered a week later is recoverable from bytes and not recoverable from text, which makes a short retention window very cheap insurance. It expires on the same schedule as every other raw payload.

How do I detect double-encoded text after the fact? #

Look for the characteristic sequences — é, ’, †— in stored fields. A pattern scan over a sample is enough to establish whether a host is affected. Do not attempt to repair in place: re-run the decode from the retained bytes where they still exist, and recrawl the affected range where they do not.

Does the same resolution order apply to JSON and XML responses? #

The order is similar but the sources differ. JSON is UTF-8 by specification unless a byte-order mark says otherwise, so honouring a contradictory Content-Type charset is usually wrong — decode as UTF-8 and record the disagreement. XML carries its own declaration in the prolog, which takes precedence over the transport header for a standalone document. In both cases the principle holds: attempt a strict decode, record which source was used, and never silently substitute replacement characters.

Should a site-specific override ever be permanent? #

Treat it as a dated decision rather than a permanent fact. Encoding problems are usually fixed eventually — a migration completes, a CDN configuration is corrected — and an override that outlives the problem starts corrupting correct content. Give each override a review date and a recorded reason, and re-test it on that date by attempting the normal resolution path against a fresh sample.