Robots.txt Beyond the Basics: Crawl-Delay, Pattern Matching, and Edge Cases

Editorial Team 2025-04-14 4 min read

The Basics Aren't Enough

Everyone knows the basics of robots.txt: User-agent, Disallow, Allow. But most robots.txt files I audit have problems that stem from misunderstanding how crawlers actually parse this file. Let's go beyond the basics.

How Crawlers Actually Parse Robots.txt

Here's something that trips people up: robots.txt is not a standard in the traditional sense. The original 1994 protocol was informal, and different crawlers implement it differently. Google formalized their interpretation in RFC 9309 (published 2022), but Bing, Yandex, and others have their own quirks.

Key parsing rules for Googlebot:

  • Most specific rule wins (by character length), not the first rule
  • Allow and Disallow are evaluated together — the longest matching path takes precedence
  • * matches any sequence of characters, including /
  • $ anchors to the end of URL
  • Rules are case-sensitive for the path portion

That "most specific wins" behavior is critical. Consider this:

User-agent: *
Disallow: /private/
Allow: /private/public-page

The Allow rule is more specific (longer path), so /private/public-page will be crawled even though its parent directory is disallowed. This works reliably on Google. On some other crawlers, the order might matter instead — which is why testing matters.

Crawl-Delay: The Misunderstood Directive

The Crawl-delay directive tells crawlers to wait N seconds between requests. Here's the critical detail: Google completely ignores Crawl-delay. They've never supported it. If you want to control Google's crawl rate, you need to use Search Console's crawl rate setting.

Bing, Yandex, and most other crawlers do respect it:

User-agent: bingbot
Crawl-delay: 10

User-agent: yandex
Crawl-delay: 5

Be careful with crawl-delay values. Setting it to 10 means 10 seconds between each request — that's only 8,640 pages per day. For a site with 100,000 pages, Bing would need almost 12 days to crawl everything, assuming continuous crawling. I'd keep crawl-delay under 2 seconds for most sites, and only use it if you're actually experiencing server load issues from bot traffic.

Pattern Matching with Wildcards

This is where robots.txt gets powerful — and dangerous. The * wildcard matches zero or more characters.

Blocking Parameter URLs

Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=

These rules block any URL containing those query parameters. Note the * before the ? — without it, you'd only block URLs where the parameter appears immediately after the domain root.

Blocking File Types

Disallow: /*.pdf$
Disallow: /*.xml$

The $ anchor ensures you're matching the end of the URL. Without it, /*.pdf would also block /documents/pdf-guide/chapter-one because the pattern matches "pdf" anywhere in the URL.

A Common Wildcard Mistake

# DON'T do this:
Disallow: /blog/*/comments

# This blocks:
# /blog/post-one/comments  ✓ (intended)
# /blog/2026/march/post/comments  ✓ (intended)
# /blog/comments  ✗ (* matches zero characters too!)

Since * matches zero or more characters, the pattern above also matches /blog/comments directly. If that's a valid page you want crawled, you'll need an explicit Allow rule for it.

Edge Cases That Catch People Off Guard

Robots.txt and Noindex Are Different Things

Disallowing a URL in robots.txt prevents crawling, not indexing. If a page has inbound links and is disallowed in robots.txt, Google might still index the URL — they just won't be able to read its content. You'll see entries in search results with "No information is available for this page" or a snippet pulled from anchor text of linking pages.

If you want to deindex something, you need to allow crawling (so Google can see the noindex tag) or use the URL Removal Tool for temporary removal. This is one of the most common misunderstandings in SEO.

The 5MB File Size Limit

Google enforces a 500KB limit on robots.txt (as per their documentation — some sources cite 500 KiB). Rules beyond this limit are ignored. I've seen enterprise sites with auto-generated robots.txt files that exceeded this limit, effectively leaving the rest of their rules unprocessed. If your file is getting large, consolidate rules using wildcards.

HTTP Status Codes for Robots.txt Itself

  • 200 — normal, rules are followed
  • 404 — Googlebot treats this as "everything is allowed"
  • 5xx — Googlebot treats this as "everything is disallowed" (temporarily) and will retry later
  • 301/302 — Googlebot follows redirects (up to 5 hops) to find the robots.txt

That 5xx behavior is worth remembering. If your server has a brief outage and robots.txt returns a 500, Googlebot will temporarily stop crawling your entire site. It's conservative by design — if it can't read the rules, it assumes everything might be blocked.

HTTPS vs HTTP

Robots.txt is protocol-specific. The rules in https://example.com/robots.txt only apply to HTTPS URLs. If you still have HTTP versions of your pages accessible (even if they redirect), you technically need a separate robots.txt on the HTTP version. In practice, this rarely matters if your HTTP-to-HTTPS redirects are working properly, but it's worth knowing.

Testing Your Robots.txt

Google Search Console has a robots.txt tester (under the legacy tools section). Use it. Paste in your robots.txt and test specific URLs to verify they're blocked or allowed as intended.

You can also test from the command line:

curl -A "Googlebot" https://example.com/robots.txt

Some CDNs serve different robots.txt files based on user agent — I've seen Cloudflare setups that accidentally blocked bots at the edge level while the origin server had correct rules. Always test with a bot user agent to see what crawlers actually receive.

Practical Template

Here's a starting point that works for most sites:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?ref=
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

I'd also recommend blocking AI training crawlers (GPTBot, CCBot, Google-Extended) unless you specifically want your content used for LLM training. These bots consume significant bandwidth on popular sites and don't drive any search traffic back to you.

Review your robots.txt quarterly. It's a small file with outsized impact — one wrong rule can deindex an entire section of your site.