Compliant Web Scraping

Detecting and Respecting noindex / nofollow Directives #

A publisher can allow a crawler to fetch a page while still signalling that the page should not be indexed, or that its outbound links should not be followed. Those signals — the noindex and nofollow robots directives — live in the HTML <meta> tag, in the X-Robots-Tag HTTP header, and in the rel attribute of individual links. Honouring them is a narrow but load-bearing part of running a defensible crawler, and it sits under the Mapping Terms of Service for Scrapers technique within the broader Compliance & Ethical Crawling Foundations section. This page covers how to detect all three carriers and wire them into link-following logic.

Problem Framing #

robots.txt controls access — whether you may request a URL at all. The noindex/nofollow family is different: the server has already served you the page, and is now expressing intent about what you do with it. noindex says “do not add this page to your index / dataset”; nofollow says “do not use the links on this page (or this specific link) to discover more URLs, and do not pass ranking signal through them.”

In production this arises constantly. A site marks its faceted-search results noindex to keep them out of search engines but still wants humans to browse them. A forum adds rel="nofollow" to user-submitted links to disavow spam. A paginated archive sets X-Robots-Tag: noindex, nofollow at the CDN layer so it never appears in the header of the served HTML you might be caching. A crawler that ignores these is not just impolite — it is disregarding an explicit, machine-readable expression of the publisher’s wishes, which weakens any argument that your collection was authorised. Because the header form and the meta form can disagree, and because directives can be scoped to a named bot, detection has to check every carrier before a link is queued.

Step-by-Step Implementation #

  1. Read the X-Robots-Tag response header first. It arrives before you parse the body and can apply noindex/nofollow to non-HTML resources (PDFs, images) that have no place for a meta tag. A header may carry multiple comma-separated tokens and may be bot-scoped as googlebot: noindex.
  2. Parse the <meta name="robots"> tag (and any bot-specific <meta name="yourbot">) from the <head>. Treat the wildcard robots name as applying to everyone.
  3. Merge header and meta directives — the union of both applies. If either says noindex, the page is not stored; if either says nofollow, no link on the page is enqueued.
  4. When the page is followable, inspect each link’s rel attribute for nofollow (and related sponsored / ugc values) and skip those individual targets.
  5. Record the decision against the URL so audits can show why a page was dropped or a link was not followed.
import re
from dataclasses import dataclass
from bs4 import BeautifulSoup

MY_BOT = "compliantbot"

@dataclass
class RobotsDirectives:
    noindex: bool = False
    nofollow: bool = False

def _tokens_for_bot(field_value: str) -> str:
    """A directive line may be 'noindex, nofollow' or 'googlebot: noindex'."""
    if ":" in field_value:
        bot, _, directives = field_value.partition(":")
        if bot.strip().lower() not in ("robots", MY_BOT):
            return ""  # scoped to a different bot — ignore
        return directives
    return field_value

def parse_directives(headers: dict, html: str) -> RobotsDirectives:
    d = RobotsDirectives()

    # 1. X-Robots-Tag header (there may be several, comma-joined by the client)
    for raw in headers.get("X-Robots-Tag", "").split(","):
        toks = _tokens_for_bot(raw).lower()
        d.noindex |= "noindex" in toks or "none" in toks
        d.nofollow |= "nofollow" in toks or "none" in toks

    # 2. <meta name="robots"> / <meta name="compliantbot">
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all("meta"):
        name = (tag.get("name") or "").lower()
        if name in ("robots", MY_BOT):
            content = (tag.get("content") or "").lower()
            d.noindex |= "noindex" in content or "none" in content
            d.nofollow |= "nofollow" in content or "none" in content
    return d

def followable_links(html: str, page_nofollow: bool) -> list[str]:
    """Yield hrefs unless the page or the individual link is nofollow."""
    if page_nofollow:
        return []  # page-level nofollow suppresses every outbound link
    soup = BeautifulSoup(html, "lxml")
    out = []
    for a in soup.find_all("a", href=True):
        rel = {r.lower() for r in (a.get("rel") or [])}
        if rel & {"nofollow", "sponsored", "ugc"}:
            continue  # link-level disavowal — do not enqueue
        out.append(a["href"])
    return out

The none token is shorthand for noindex, nofollow; treating it as both is required for correctness. Note that noindex governs whether you retain the page in your dataset, while nofollow governs whether you expand its links — they are independent and a page can carry either, both, or neither.

Verification & Testing #

Exercise both carriers and the scoped-bot case. httpbin.org/response-headers lets you mint an arbitrary X-Robots-Tag for an integration check.

# Confirm the header is read: this should be classified noindex+nofollow.
curl -s -D - "https://httpbin.org/response-headers?X-Robots-Tag=noindex,%20nofollow" -o /dev/null | grep -i x-robots-tag
def test_meta_nofollow_suppresses_links():
    html = '<html><head><meta name="robots" content="nofollow"></head>' \
           '<body><a href="/a">a</a></body></html>'
    d = parse_directives({}, html)
    assert d.nofollow and not d.noindex
    assert followable_links(html, d.nofollow) == []

def test_header_beats_missing_meta():
    d = parse_directives({"X-Robots-Tag": "noindex"}, "<html></html>")
    assert d.noindex and not d.nofollow

def test_rel_nofollow_skips_single_link():
    html = '<a href="/keep">k</a><a href="/spam" rel="nofollow">s</a>'
    assert followable_links(html, page_nofollow=False) == ["/keep"]

def test_directive_scoped_to_other_bot_ignored():
    d = parse_directives({"X-Robots-Tag": "googlebot: noindex"}, "<html></html>")
    assert not d.noindex  # not addressed to us

Compliance & Operational Guardrails #

  • Directives are intent, not access control. Fetching a noindex page is not a breach by itself, but retaining or republishing it after being told not to index undercuts any claim that your collection respected the publisher’s expressed wishes.
  • The union rule fails safe. When the header and meta tag disagree, apply the stricter of the two. Never let a body-level index override a header-level noindex.
  • Respect link-level nofollow at enqueue time, not after the fetch. Discovering and requesting a nofollow target and only then discarding it wastes the target’s bandwidth and defeats the purpose of the signal.
  • Log every drop. A record of “page X: noindex via X-Robots-Tag, not stored” is exactly the evidence you want in an audit; feed it into your documenting scraping authorization and data lineage trail.

Common Mistakes #

  1. Checking only the meta tag. CDNs and app servers frequently set X-Robots-Tag on responses whose HTML carries no robots meta at all — and on non-HTML files that cannot carry one. Header-only noindex is common.
  2. Treating page-level nofollow and link-level rel="nofollow" as the same switch. A page can be fully followable while individual spammy links are disavowed; conversely a page nofollow suppresses every link regardless of rel.
  3. Ignoring bot-scoped directives. X-Robots-Tag: googlebot: noindex addresses Googlebot, not you — but robots: noindex and a directive naming your own agent both do. Parse the optional bot: prefix instead of matching substrings blindly.

Frequently Asked Questions #

Is ignoring a noindex directive illegal? #

Not inherently — noindex is an indexing preference, not an access-control mechanism, so fetching the page is not unauthorised access. The risk is downstream: if a publisher told you not to index a page and you retained and republished it anyway, you have discarded a clear, machine-readable signal of their wishes, which weakens your compliance and ToS position and can support a misuse claim.

Which wins when the X-Robots-Tag header and the meta tag disagree? #

Apply the union and take the stricter outcome. If the header says noindex and the meta tag omits it, the page is still noindex. Search engines resolve conflicts toward the most restrictive directive, and a crawler that wants to demonstrate good faith should do the same rather than cherry-picking the permissive source.

Does nofollow stop me from crawling a URL entirely? #

No. nofollow is a discovery and endorsement signal: it tells you not to use this link to reach a target or to pass ranking value. If you reach the same URL through another followable path, or it is allowed by the target’s own robots.txt, you may still crawl it. nofollow constrains the edge in the link graph, not the destination node.