Compliant Web Scraping

Setting a From Header and Contact URL So Operators Can Reach You #

The single fastest way to turn a hostile block into a polite email is to make your crawler reachable. This page covers the two mechanisms that do that: the RFC 9110 From request header carrying a real mailbox, and a contact URL embedded inside your User-Agent string. It is a detail page under ethical User-Agent configuration, part of the broader Compliance & Ethical Crawling Foundations section, and it assumes you have already settled on a stable agent name and are now adding the identity fields that let a site operator contact you before they escalate to an IP ban or an abuse report.

Problem Framing #

When a site operator sees unfamiliar traffic in their logs, they have two choices: investigate or block. An anonymous python-requests/2.31.0 user agent gives them nothing to investigate, so they block — and blocks are broad, permanent, and often extend to your whole subnet or cloud provider. A request that carries a named crawler, a monitored mailbox, and a URL explaining who you are converts that same log line into a low-cost conversation. Operators regularly email crawler owners to negotiate a rate, request a delay window, or point you at an official API instead of pulling their site.

This matters most for pipelines that run unattended for weeks. By the time a complaint would surface, the person who wrote the scraper may have moved on, the ticket may be stale, and the only durable record of intent is what you put on the wire at request time. Making yourself reachable is both an ethical baseline and a legal-defensibility asset: it demonstrates good faith, which is exactly the posture you want on record if a Terms of Service dispute ever arises.

Step-by-Step Implementation #

  1. Register a dedicated, monitored mailbox. Use a role address such as [email protected], not a personal inbox that will bounce when someone leaves. Route it to a ticketing queue that a human reads within one business day.
  2. Publish a contact page at a stable URL (for example https://yourcompany.example/bot). This is what you embed in the agent string; the next section covers what belongs on it.
  3. Set the From header to a bare email address. RFC 9110 defines From as a single mailbox production — an addr-spec, no display name, no angle brackets. Send [email protected], not "Acme Bot" <[email protected]>.
  4. Embed the contact URL in the User-Agent using the conventional (+https://...) comment syntax that Googlebot and other well-behaved crawlers popularised. The leading + is a de-facto marker that the parenthetical is a reference URL.
  5. Send both on every request by binding them to a session so no code path can accidentally omit them.
import requests

CONTACT_EMAIL = "[email protected]"
CONTACT_URL = "https://yourcompany.example/bot"

session = requests.Session()
session.headers.update({
    # User-Agent: a stable product token, version, and a +contact URL comment.
    "User-Agent": f"AcmeResearchBot/1.2 (+{CONTACT_URL})",
    # From: a single bare mailbox per RFC 9110 sec. 10.1.2 — no display name.
    "From": CONTACT_EMAIL,
})

# Every request through this session now carries the identity headers,
# so an operator can reach a human before deciding to block the IP range.
resp = session.get("https://target.example/catalog", timeout=10)
resp.raise_for_status()

For a Go client the same discipline applies — set the headers once on a shared http.Client transport wrapper:

req, _ := http.NewRequest("GET", "https://target.example/catalog", nil)
req.Header.Set("User-Agent", "AcmeResearchBot/1.2 (+https://yourcompany.example/bot)")
req.Header.Set("From", "[email protected]") // bare mailbox, RFC 9110

What to put at the contact URL #

The page should let an operator decide, in under thirty seconds, whether your traffic is acceptable. Include:

  • Who runs the crawler — organisation name and purpose in one sentence.
  • What it collects and why — the data categories and the use case.
  • How to reach a human — the same mailbox as your From header.
  • How to opt out or throttle — the robots.txt groups you honour and an invitation to email for a custom Crawl-delay.
  • Your source IP ranges or reverse-DNS pattern, so operators can verify the traffic is genuinely yours and not a spoofer.

Verification & Testing #

Confirm both fields actually leave your process — do not trust that you set them, prove it. httpbin echoes request headers back to you:

curl -s https://httpbin.org/headers \
  -H "User-Agent: AcmeResearchBot/1.2 (+https://yourcompany.example/bot)" \
  -H "From: [email protected]" | jq '.headers'

For the Python session, assert against the same echo endpoint in a unit test so a future refactor cannot silently drop the headers:

def test_identity_headers_are_sent():
    r = session.get("https://httpbin.org/headers", timeout=10).json()
    hdrs = r["headers"]
    assert hdrs["From"] == "[email protected]"
    assert "+https://yourcompany.example/bot" in hdrs["User-Agent"]

Finally, send a real message to the contact mailbox from an outside account and confirm a ticket is created. An address that bounces is worse than no address, because it signals reachability that does not exist.

Compliance & Operational Guardrails #

  • Identity headers are not personal data about the target. The From mailbox is your contact point, so it raises no GDPR concern about the site you crawl — but keep it a role address, not a named individual, to avoid creating an unnecessary personal-data footprint of your own staff.
  • Reachability supports a good-faith defence. Under authorisation-based frameworks like the CFAA, demonstrable transparency and a working opt-out channel weaken any claim that access was unauthorised or covert.
  • Keep the URL stable across versions. Bump the version token in the User-Agent freely, but never move the contact URL without a redirect; operators bookmark it from historical logs.
  • Honour what the page promises. If the contact page says you respect Crawl-delay, your pacing must actually enforce it — pair this identity work with real polite rate limiting.

Common Mistakes #

  1. Putting a display name in From. Acme Bot <crawler@...> violates the header’s single-mailbox grammar and some strict proxies will strip or reject it. Send the bare addr-spec.
  2. Using an unmonitored inbox. A contact address that no human reads defeats the entire purpose; operators email once, get silence, and block.
  3. Rotating the contact URL with every deploy. Identity fields must be the stable anchor while other attributes vary; churning them makes historical log correlation impossible.

Frequently Asked Questions #

Is the From header still valid in modern HTTP? #

Yes. RFC 9110 (the current HTTP semantics specification) retains From and defines its value as a single mailbox. It is optional and rarely sent by browsers, which is exactly why it reads as a deliberate identity signal from an automated client. Servers may log it but must not use it for access control by default.

Should I put the contact URL in User-Agent, the From header, or both? #

Both, because they are read in different places. The +URL comment in User-Agent appears in access logs that operators grep by hand, while From is a structured field their tooling can extract programmatically. Sending both maximises the chance the right person sees a way to reach you.

Will identifying my crawler get it blocked faster? #

Occasionally a site blocks all non-browser agents regardless, but that is uncommon and usually reversible by email — which only works if you were reachable. Anonymous traffic that trips a heuristic gets blocked with no recourse at all, so honest identification is the lower-risk posture over any real time horizon.