robots.txt and Web Scraping: How to Check Whether a Site Allows Scraping

Every well-run website publishes a short instruction sheet for bots at /robots.txt, and it spells out which sections automated visitors are asked to leave alone. Opening that sheet is where any scraping assessment should begin, yet it settles only part of the question. The file expresses a wish rather than enforcing anything, it grants no rights, and the operator's terms, its tolerance for request volume and the legal status of the data weigh just as heavily on the final decision.

Below we go through the file line by line, with extra care for the parts people misread most often: which rule wins when two overlap, how wildcards behave, what a 404 or 503 on the file means and which directives Google simply skips. You will also see a Python helper for testing addresses, how publishers single out AI crawlers in 2026, the other checks worth doing before the first request, and a reusable checklist at the end.

What robots.txt Is and What It Is Not

The convention dates back to 1994; after almost three decades as an informal agreement it was written down by the IETF as RFC 9309 in September 2022. A crawler looks for it only in the root folder, and each copy governs a single origin, meaning one particular scheme, hostname and port. So shop.example.com keeps its own rules, independent of whatever www.example.com says.

Things the file is good for:

• naming URL paths that one specific bot, or bots in general, are asked to skip;
• pointing to sitemap files, which double as a ready list of pages the owner wants found;
• occasionally passing along informal hints, a crawl delay being the classic example.

Things people wrongly expect from it:

• enforcement: a program that never reads the file can still pull every page, since nothing on the server is locked;
• permission: an Allow line grants nothing beyond what the terms, copyright and privacy rules already permit;
• search visibility: keeping a URL out of Google results is the job of a noindex meta element or an X-Robots-Tag response header.

How to Read the File

Internally the document is a series of blocks. A block opens with at least one User-agent line naming the bots it addresses, followed by path rules that say Allow or Disallow. The sample below mixes the patterns you meet most often:

User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /*?sessionid=
Allow: /search/help
Crawl-delay: 5

User-agent: GPTBot
Disallow: /

User-agent: PartnerFeedBot
Allow: /

Sitemap: https://www.example.com/sitemap-index.xml

Four points cause most misreadings:

1. A bot follows exactly one block. It picks the block whose name matches it best and then disregards the asterisk block completely. That is why PartnerFeedBot in the sample may open cart pages: its dedicated block permits everything.
2. Specificity beats position. If an Allow and a Disallow both fit a URL, RFC 9309 applies whichever pattern is longer, and an Allow takes a tie. Where the lines sit in the file is irrelevant.
3. A dollar sign pins the end. With Disallow: /*.pdf$, an address that finishes with .pdf is excluded, yet /file.pdf?v=2 is not.
4. Paths respect letter case, while bot names are compared case-insensitively.

Directives Outside the Standard

RFC 9309 says nothing about Crawl-delay. Google disregards the line, whereas engines such as Bing and Yandex have historically taken it into account. For your own collector the number is still valuable, because it reveals the tempo the operator is comfortable with, and following it costs almost nothing. Clean-param and Host belong to Yandex and can safely be skipped by a general-purpose scraper.

When the File Is Missing or Broken

The RFC also covers failures while fetching the file itself:

• a 4xx status such as 404 means the rules are unavailable, so a bot may treat the host as unrestricted;
• a 5xx status or a dropped connection means the rules are unreachable, so a bot should behave as if every path were excluded until a later attempt succeeds;
• caching of a fetched copy should not exceed a day, apart from periods when a fresh copy cannot be obtained.

Parsers must also cope with files of at least 500 KiB. Google treats that figure as a hard ceiling and disregards anything past it.

How to Check robots.txt in Python

Python includes a ready-made parser in its standard library. The helper below reports whether your bot may request an address, plus any crawl delay and the listed sitemaps:

from urllib.robotparser import RobotFileParser
from urllib.parse import urlsplit

BOT_NAME = "AcmePriceWatch/1.2 (+https://acme.example/bot)"
BOT_TOKEN = "AcmePriceWatch"

def robots_verdict(page_url: str) -> dict:
    parts = urlsplit(page_url)
    rules = RobotFileParser()
    rules.set_url(f"{parts.scheme}://{parts.netloc}/robots.txt")
    rules.read()
    return {
        "allowed": rules.can_fetch(BOT_TOKEN, page_url),
        "delay": rules.crawl_delay(BOT_TOKEN),
        "sitemaps": rules.site_maps(),
    }

print(robots_verdict("https://books.toscrape.com/catalogue/page-2.html"))

Put the long BOT_NAME into your User-Agent header, so administrators can tell who is visiting and how to reach you, and evaluate rules against the short token. One caveat: the built-in module walks the rules top to bottom instead of applying the longest-pattern principle from RFC 9309, so a file with overlapping Allow and Disallow entries may get a different verdict. Scrapy ships with Protego as its default parser, which is designed to track the current specification more faithfully, and a dedicated library like that is the safer option for production crawlers.

AI Crawlers and robots.txt in 2026

From 2023 onward a growing number of publishers have written separate blocks aimed at AI bots. Tokens that turn up frequently are GPTBot and OAI-SearchBot from OpenAI, Anthropic's ClaudeBot, Google-Extended (a switch for Google's AI products rather than a distinct crawler), Applebot-Extended, CCBot from Common Crawl, PerplexityBot and Bytespider. Vendors rename agents and introduce new ones, so consult each company's own crawler page instead of copying lists from older posts.

Two further developments are worth tracking. In 2025 Cloudflare proposed a Content-Signal line through which a site can say, for instance, yes to search indexing and no to model training; it records a preference and sits outside RFC 9309. Separately, an IETF group is drafting shared terms for AI usage preferences. The practical rules stay the same: when you gather pages for a training corpus or a retrieval system, run your bot under its own name, obey blocks addressed to AI agents and count declared training preferences among the site's conditions. How such pipelines are put together is described in our article on AI web scraping.

Beyond robots.txt: What Else to Check

Law in this area differs between countries and keeps shifting. Within the EU, the text and data mining exception lets rightsholders opt out by machine-readable means, and robots.txt is frequently named as one of them. American courts have reached mixed, fact-dependent conclusions on collecting public pages. Disregarding the file does not by itself make a project unlawful, and obeying it does not by itself make a project lawful. Anything commercial deserves a review by a lawyer familiar with the relevant jurisdiction. For the technical side of the blocks you may run into, see our guide on errors when parsing websites.

Pre-Scrape Checklist

1. Download /robots.txt from every host involved, subdomains included, and find the block your bot falls under.
2. Run your real URL patterns, query strings and all, through a parser.
3. Search the terms for wording about bots, crawling, scraping or automated access.
4. Find out whether an official API, bulk export or data licence exists.
5. Make sure the pages render for a visitor who is not signed in.
6. Note every kind of personal data on the pages and drop what you do not need.
7. Start slow, follow any crawl delay and Retry-After value, and run large jobs when traffic is low.
8. Announce your bot through a descriptive User-Agent that includes a contact link.
9. During long runs, fetch the rules again at least every 24 hours.
10. Record what you verified and on which date.

Where Proxies Fit In

A proxy never widens what an operator permits. In a well-behaved setup it solves practical problems: it spreads allowed traffic so no single address strains the server, lets you see pages the way visitors in a chosen country see them, and keeps lengthy jobs from collapsing on one IP. Open data that a site lets you collect usually works fine through datacenter proxies, while consumer platforms whose content varies by region respond better to residential addresses. Pointing a pool of IPs at paths the file excludes is precisely the behaviour this checklist exists to rule out. For configuration details, read our guide to proxies in web scraping and the broader introduction to web scraping.