robots.txt

How robots.txt works

Before crawling an origin, a compliant web crawler normally requests /robots.txt, such as https://example.com/robots.txt. The file groups path rules under one or more crawler identities. A crawler selects the group matching its identity, then checks each requested URL against that group.

User-agent: *
Disallow: /account/
Disallow: /checkout/
Allow: /account/help/
Sitemap: https://example.com/sitemap.xml

These rules ask all matching crawlers to avoid /account/ and /checkout/, while permitting /account/help/. When rules overlap, the more specific path match wins.

Common directives

• User-agent: names the crawler product that the following rules address. An asterisk matches crawlers without a more specific group.
• Disallow: identifies a path the crawler shouldn’t request. An empty value doesn’t block anything.
• Allow: creates an exception inside a broader blocked path.
• Sitemap: points to a sitemap for URL discovery. It doesn’t grant permission to crawl those URLs.

Some files contain Crawl-delay, but it isn’t standardized and crawler support varies.

Why it matters for scraping

robots.txt communicates the site owner’s crawling preferences. It doesn’t enforce access by itself, so cooperative clients must choose to follow it. Check it before starting web scraping and whenever you add a new origin.

A rotating proxy changes the IP address a website sees. It doesn’t change robots.txt rules or make a disallowed path acceptable to crawl. Rules can differ by crawler identity, so don’t claim another identity to receive a less restrictive policy.

How to crawl responsibly

1. Fetch the root file. Each scheme, host and port can have its own rules, including separate subdomains.
2. Use a proper parser. Path case, encoded characters and overlapping rules can affect the result.
3. Exclude disallowed URLs. Don’t use IP rotation to work around a published restriction.
4. Control your request rate. For allowed paths, keep concurrency reasonable and slow down when rate limiting or a 429 response appears.
5. Follow other restrictions. robots.txt neither replaces site terms nor grants access to protected content.

The file is public and isn’t a security control. Sensitive pages still need authentication and server-side authorization.

FAQ

Where should robots.txt be located?

It should be available at the exact root path, such as https://example.com/robots.txt. A file at /folder/robots.txt doesn’t control crawling for the site.

Does robots.txt block web scraping?

Not technically. It is a voluntary protocol that compliant crawlers follow, not a firewall or access-control system.

Does Disallow prevent a page from appearing in search results?

Not reliably. Disallow controls crawling, not indexing, and a search engine may discover the URL through links without fetching its content.