Compliant Web Scraping

Validating Scraped Dates and Currency Values with Pydantic v2 #

Scraped date strings and price fields are where type discipline usually breaks down: a listing page emits "Jul 3, 2026", "03/07/2026", and "hace 2 días" on the same crawl, and a product feed prints "€1.299,00" next to "$1,299.00". This page is a focused recipe for coercing those messy values into datetime and Decimal fields safely, using Pydantic v2 field_validator hooks. It sits under Schema Validation with Pydantic within the broader Data Parsing & Transformation Pipelines section, and assumes you already validate the surrounding record structure — here we drill into the two field types that cause the most silent corruption.

Problem Framing #

Dates and money share a dangerous property: the wrong parse still looks plausible. 03/07/2026 is 3 July in most of the world and 7 March in the US; nothing in the string tells you which, so a naive parser will happily produce a valid-but-wrong datetime. Money is worse, because floating-point arithmetic silently loses precision — 0.1 + 0.2 != 0.3 — and thousands separators differ by locale, so 1.299 is either “one thousand two hundred ninety-nine” (German) or “one point two nine nine” (US). If either error reaches your warehouse, downstream aggregates, tax calculations, and time-window queries are quietly wrong and nobody sees an exception.

The fix is to make the parsing rules explicit at the validation boundary, store money as Decimal, store timestamps as timezone-aware UTC, and reject anything the rules cannot resolve rather than guessing.

Step-by-Step Implementation #

  1. Model money as Decimal, never float. Annotate the field as Decimal and use a field_validator(mode="before") to strip currency symbols and normalise separators before Pydantic coerces the value.
  2. Detect the decimal convention from the string shape rather than assuming the site’s locale — the last . or , that is followed by exactly two digits is the decimal separator; everything else is a grouping separator to delete.
  3. Parse dates with dateutil but pass dayfirst/yearfirst explicitly per source, so ambiguous dd/mm vs mm/dd strings resolve deterministically instead of by library default.
  4. Normalise every timestamp to UTC. Attach the source site’s timezone when the string is naive, then convert with astimezone(timezone.utc) so all stored times are comparable.
  5. Fail loudly on unparseable input by raising ValueError inside the validator — Pydantic wraps it in a ValidationError your dead-letter routing already handles.
# money_and_dates.py
from decimal import Decimal, InvalidOperation
from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import re

from dateutil import parser as dateutil_parser
from pydantic import BaseModel, field_validator, Field, ValidationInfo

_CURRENCY_SYMBOLS = "$€£¥₹"
# Decimal separator = a '.' or ',' followed by exactly 2 digits at end of number.
_DECIMAL_TAIL = re.compile(r"[.,](\d{2})\b")


def _to_decimal(raw: str) -> Decimal:
    s = raw.strip()
    for sym in _CURRENCY_SYMBOLS:
        s = s.replace(sym, "")
    s = re.sub(r"[A-Za-z\s ]", "", s)  # drop 'USD', NBSP, stray letters
    if not s:
        raise ValueError("empty money string")
    # Identify the decimal separator by the two-digit tail; delete all others.
    m = _DECIMAL_TAIL.search(s)
    if m:
        dec_sep = s[m.start()]
        s = s.replace("." if dec_sep == "," else ",", "")  # kill grouping
        s = s.replace(dec_sep, ".")
    else:
        s = s.replace(",", "").replace(".", "")  # integer amount, no decimals
    try:
        return Decimal(s)
    except InvalidOperation as exc:
        raise ValueError(f"unparseable money value: {raw!r}") from exc


class ScrapedListing(BaseModel):
    price: Decimal = Field(gt=0, description="Money as Decimal, never float")
    currency: str = Field(pattern=r"^[A-Z]{3}$")  # ISO 4217
    published_at: datetime  # stored timezone-aware, UTC
    source_tz: str = Field(default="UTC", exclude=True)  # e.g. "America/New_York"
    dayfirst: bool = Field(default=False, exclude=True)  # per-source hint

    @field_validator("price", mode="before")
    @classmethod
    def parse_price(cls, v: object) -> object:
        return _to_decimal(v) if isinstance(v, str) else v

    @field_validator("published_at", mode="before")
    @classmethod
    def parse_published_at(cls, v: object, info: ValidationInfo) -> object:
        if not isinstance(v, str):
            return v
        try:
            dt = dateutil_parser.parse(v, dayfirst=info.data.get("dayfirst", False))
        except (ValueError, OverflowError) as exc:
            raise ValueError(f"unparseable date: {v!r}") from exc
        if dt.tzinfo is None:  # naive -> attach source tz, then normalise to UTC
            tz = ZoneInfo(info.data.get("source_tz", "UTC"))
            dt = dt.replace(tzinfo=tz)
        return dt.astimezone(timezone.utc)

Because field_validator runs in field-declaration order, source_tz and dayfirst are populated before published_at and are available on info.data. Keep those hint fields per source rather than global, and store them alongside your selector config so a US site and a German site can share the same model class with different parsing behaviour.

Verification & Testing #

Assert on concrete ambiguous inputs — this is the only reliable way to prove the locale logic holds. Test both separator conventions and both date orderings.

# test_money_and_dates.py
from decimal import Decimal
from datetime import timezone
import pytest
from money_and_dates import ScrapedListing, _to_decimal


@pytest.mark.parametrize("raw,expected", [
    ("$1,299.00", Decimal("1299.00")),   # US grouping
    ("€1.299,00", Decimal("1299.00")),   # German grouping
    ("£49.99", Decimal("49.99")),
    ("1 234,50 USD", Decimal("1234.50")),  # NBSP grouping
])
def test_money_locale(raw, expected):
    assert _to_decimal(raw) == expected


def test_date_dayfirst_disambiguation():
    us = ScrapedListing(price="1", currency="USD",
                        published_at="03/07/2026", dayfirst=False)
    eu = ScrapedListing(price="1", currency="EUR",
                        published_at="03/07/2026", dayfirst=True)
    assert us.published_at.month == 3   # March
    assert eu.published_at.month == 7   # July
    assert us.published_at.tzinfo == timezone.utc


def test_unparseable_rejected():
    with pytest.raises(Exception):
        ScrapedListing(price="free", currency="USD", published_at="2026-01-01")

Run pytest -q in CI as a gate before deploying any selector change. Because Decimal("1299.00") == Decimal("1299.0"), compare with the exact scale you expect, or normalise with .quantize() when the scale itself matters for storage.

Compliance & Operational Guardrails #

  • Store money as Decimal end to end — write it to a NUMERIC/DECIMAL column, not FLOAT. If a price feeds billing, invoicing, or tax reporting, float rounding is an auditable financial error, not a cosmetic one.
  • Record the parse decision, not just the result. Log the raw string, the resolved dayfirst, and the applied source_tz alongside the record so data lineage stays defensible; this pairs with the audit-log patterns in the parent Schema Validation with Pydantic guide.
  • Timestamps that carry personal data (account activity, review dates) fall under data-minimisation rules — normalise them to UTC but do not enrich beyond what the source published, and hash or drop any co-located PII at the same validation boundary before persistence.
  • Reject rather than guess on ambiguous input. A quarantined record in a dead-letter queue is recoverable; a wrong-but-valid date silently poisoning a time-series is not.

Common Mistakes #

  1. Using float for money. It looks fine in tests with round numbers and corrupts sums at scale. Always Decimal, and strip separators before construction — Decimal("1,299") raises InvalidOperation.
  2. Trusting dateutil’s default dayfirst=False for European sources. The default silently turns 3 July into 7 March. Set the flag per source and test both orderings.
  3. Storing naive datetimes. A timestamp without a timezone is only meaningful next to the machine that made it. Attach the source zone, convert to UTC, and store aware datetime objects.

Frequently Asked Questions #

Why not use Pydantic’s built-in condecimal or AwareDatetime types instead of custom validators? #

Those constrain the value after it is already a Decimal or datetime, but they cannot clean a "€1.299,00" string or disambiguate 03/07/2026 — coercion fails before the constraint runs. Use a mode="before" validator to normalise the raw string first, then layer AwareDatetime or a condecimal(gt=0) constraint on top for the final guard.

How do I handle relative dates like “2 days ago” or “yesterday”? #

dateutil cannot parse those. Add a small pre-processing branch in the before validator that matches known relative phrases per language and subtracts a timedelta from a captured crawl timestamp. Always anchor to the crawl time, not datetime.now() at validation time, so re-processing an old batch yields the same absolute date.

Should currency conversion happen inside the validator? #

No. Validation should record the amount and its original ISO 4217 currency exactly as published. FX conversion is a separate transformation stage with its own dated rate table, and folding it into validation destroys the audit trail linking the stored value to the source page.