Block AI crawlers: robots.txt, server rules and verification

To block AI crawlers, list each AI user agent in robots.txt with Disallow: /, then enforce it: return 403 to those user agents at the server or CDN, verify that a request claiming to be a known bot comes from that vendor’s published IPs or DNS, and rate-limit what is left. robots.txt stops the crawlers that choose to follow it. The rest needs enforcement and verification.

The trouble starts with the logs. A publisher sees GPTBot, ClaudeBot, Bytespider and a dozen other names hitting article pages, the origin load climbs, and none of it shows up as readers. So they paste a block list from a forum into robots.txt and hope.

That list usually blocks the wrong things. It stops a search crawler that would have cited the site, misses the fetchers that act for a live user, and does nothing about a crawler that sends a browser user agent from a home IP. Sorting the bots by what they do with a page fixes most of that. The rest is a network problem: which networks a request can come from, and why an IP block then lands on real visitors.

How to block AI crawlers, layer by layer

Three layers do three different jobs.

• robots.txt asks. RFC 9309 (September 2022) standardised the file and states that its rules are not a form of access authorization, so a compliant crawler reads it and stays out while nothing technical stops one that does not.
• The server or edge enforces. A rule in nginx, Apache or a CDN returns 403 to a matching request, and it works on anything that identifies itself, compliant or not.
• Verification decides who is real. A user agent string is text anyone can send, so checking the source IP against the vendor’s published list, its reverse DNS or a cryptographic signature is what separates the real GPTBot from a scraper wearing its name.

The AI crawler list, by purpose

Before writing a single rule, sort the bots by what they do with a page. Blocking a training crawler costs nothing in search. Blocking a search crawler removes the site from that assistant’s cited answers. Blocking a user-triggered fetcher breaks the case where a person pastes your URL into a chat.

Token	Operator	Job	robots.txt behaviour (vendor docs)
GPTBot	OpenAI	Training	Follows it
OAI-SearchBot	OpenAI	ChatGPT search index	Follows it
ChatGPT-User	OpenAI	Fetch for a live user	Rules may not apply
ClaudeBot	Anthropic	Training	Follows it
Claude-SearchBot	Anthropic	Search quality	Follows it
Claude-User	Anthropic	Fetch for a live user	Follows it
PerplexityBot	Perplexity	Answer index	Follows it
Perplexity-User	Perplexity	Fetch for a live user	Generally ignores it
Meta-ExternalAgent	Meta	Training and indexing	Follows it
Meta-ExternalFetcher	Meta	Fetch for a live user	May bypass it
CCBot	Common Crawl	Open corpus used for training	Follows it
Bytespider	ByteDance	Search and training	Says it follows it
Google-Extended	Google	Token only: Gemini training and grounding	No crawler of its own
Applebot-Extended	Apple	Token only: Apple model training	No crawler of its own

Sources for the table: OpenAI’s crawler overview (checked September 2026) says robots.txt rules may not apply to ChatGPT-User because a user starts the action. Anthropic’s crawler article (checked September 2026) lists ClaudeBot, Claude-User and Claude-SearchBot and says each honours robots.txt. Perplexity’s crawler docs (checked September 2026) and Meta’s web crawler docs (checked September 2026) carve out their user-triggered fetchers. Google-Extended and Applebot-Extended never fetch anything; they tell Googlebot’s and Applebot’s owners what they may train on.

Training, search and user fetchers

The split gives a simple default for most sites. Training crawlers take the page and send nobody back. Search crawlers take the page and may send a click with a citation. User fetchers only arrive because a person asked about your page. The operator side of the same split is covered in proxies for AI training data for training crawls and proxies for AI agents for fetches made for a person.

The ratio of crawling to clicks makes the case in numbers.

Both crawl ratios come from Cloudflare’s crawl-to-click analysis (August 2025). The domain count is from Cloudflare’s Content Signals Policy post (September 2025). Ratios move month to month as assistants add search features, so treat them as the shape of the trade, not a constant.

robots.txt rules for AI crawlers

One group per token is the clearest form, and every compliant parser reads it the same way. This file blocks the training crawlers and the two training tokens, and leaves search and user fetchers alone:

# Training crawlers and training-only tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including search engines and AI search
User-agent: *
Allow: /

Many copied lists still carry anthropic-ai and Claude-Web. Anthropic’s current article names neither, so they add nothing, though they do no harm.

RFC 9309 allows several User-agent lines above one set of rules, which keeps the file short. A bot that matches a named group ignores the * group entirely, so if the site also disallows /cart/ or /admin/ for everyone, repeat those lines inside the AI group.

To block AI bots that power search as well, add OAI-SearchBot, Claude-SearchBot and PerplexityBot to the first group. Weigh that against the citations those assistants send (see the SEO section below).

Three details catch people out when they block AI crawlers this way:

• Each subdomain needs its own file. blog.example.com/robots.txt does not cover shop.example.com. Anthropic’s article says this directly.
• Changes take about a day. OpenAI says its search systems adjust roughly 24 hours after a robots.txt change, and RFC 9309 lets crawlers cache the file for up to 24 hours.
• Keep robots.txt reachable. RFC 9309 tells a crawler that gets a 5xx on robots.txt to assume a full disallow, and one that gets a 4xx to assume no rules at all. A WAF rule that returns 403 on /robots.txt to bots turns a block into an invitation.

Blocking AI bots at the server

robots.txt will not stop a crawler that ignores it. A server rule will, as long as the bot still sends its real name. Keep /robots.txt outside the rule so compliant bots can still read the opt-out.

nginx:

map $http_user_agent $ai_training_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) 1;
}

server {
    # ... your existing config ...

    location = /robots.txt {
        # served to everyone, so compliant crawlers can read the opt-out
        try_files $uri =404;
        # if your app generates robots.txt, use its proxy_pass line here instead
    }

    location / {
        if ($ai_training_bot) { return 403; }
        # ... your existing config ...
    }
}

Apache (.htaccess with mod_rewrite):

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) [NC]
RewriteRule ^ - [F]

Google-Extended and Applebot-Extended do not belong in these rules. They never appear as user agents; Googlebot and Applebot do the fetching, and blocking those removes the site from Google Search and Apple’s search features.

Blocking a vendor’s IP range looks tidier and works worse. Anthropic warns that an IP block may not give a lasting opt-out, because it stops the crawler from reading the robots.txt that holds the opt-out. Match on user agent for declared bots, and keep IP ranges for verification.

Verify crawlers before trusting them

Any script can send User-Agent: GPTBot. Before a rule lets a request through because of its name, or a report counts it as a vendor’s traffic, check that the source matches the vendor.

Reverse and forward DNS. Google documents the method for its crawlers in Verify requests from Google crawlers and fetchers (checked September 2026): look up the IP’s hostname, confirm it ends in googlebot.com, google.com or googleusercontent.com, then resolve that hostname and confirm it returns the same IP.

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Published IP lists. OpenAI publishes the ranges each of its bots uses, linked from its crawler overview. At volume, match the source IP against those lists instead of running DNS lookups per request.

Signatures. Web Bot Auth lets a bot sign each request with HTTP Message Signatures, so a site or CDN can check the key instead of the IP. Cloudflare’s verified bot policy (updated July 2026) accepts it next to published IP lists and reverse DNS. A signature works from any address, which suits agents that run in many places.

CDN and WAF controls

Most sites behind a CDN can skip the config files. Cloudflare’s changelog of July 1, 2026 groups AI traffic into three categories, on every plan including Free:

• Search: crawlers that index content to answer questions about it later.
• Agent: automated activity acting in real time for a person, such as chat fetch bots and browser agents.
• Training: crawlers that take content to train or fine-tune a model.

Each category can be blocked on all pages, blocked only on pages that show ads, or allowed. From September 15, 2026, new domains start with Training and Agent blocked on ad pages and Search allowed, and crawlers that do both search and training fall under the Training block. A site that joined after that date may already block AI crawlers on part of its pages without anyone writing a rule, which is worth checking before adding more.

Cloudflare also offers a managed robots.txt that writes the AI groups for you, and, in closed beta, a third answer between allow and block: charging crawlers per page through pay per crawl. On WordPress and similar platforms, plugins and host panels write the same robots.txt groups and user-agent rules; check the output file rather than the plugin’s label. Other CDNs expose user-agent and bot-category rules in their WAF; the three layers above map onto whatever the provider calls them.

Content signals beyond Disallow

Disallow can only say “do not fetch”. Two newer signals let a site say “fetch for search, do not train”.

Content-Signal. Cloudflare’s policy adds a line to robots.txt with three switches: search, ai-input (use as input to a generated answer) and ai-train.

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Content-Usage. The IETF AI Preferences working group’s attachment draft (published September 14, 2026) defines a Content-Usage rule in robots.txt and an HTTP header of the same name, such as Content-Usage: train-ai=n. It is an Internet-Draft, not a finished standard.

noai meta tags. Some guides add <meta name="robots" content="noai, noimageai"> or an X-Robots-Tag: noai header. None of the OpenAI, Anthropic, Google, Apple, Perplexity or Meta crawler pages cited in this guide lists those values, so a site that relies on them has stated a wish that the named crawlers do not document reading.

Both are preferences, the same as Disallow. They cost two lines and record what the site wants, which matters when a vendor, a court or a regulator later asks whether the site opted out.

Crawlers that ignore robots.txt

The hard cases are the bots that drop their name. Cloudflare’s report on undeclared crawlers (August 4, 2025) describes a crawler that started with its declared user agent, then, when blocked, switched to a generic browser user agent, used IPs outside its published ranges and rotated across different ASNs. Cloudflare removed it from its verified bot list. Perplexity disputed the findings.

A crawler running from a cloud host sits in a hosting ASN, and a rule that challenges hosting ranges catches much of it; datacenter proxies and cloud VMs are classified on sight. A crawler routed through home connections is harder. A residential proxy exit is a real consumer IP that a real household may use an hour later, so blocking it by address blocks people.

What works against undeclared crawlers is behaviour, since the address and the name can both be borrowed:

• Rate limits per IP and per session, with a lower ceiling on article and listing URLs than on assets.
• Request shape: no JavaScript execution, no image or CSS fetches, perfectly regular timing, sequential URL walks.
• Challenges on anomalies only, so readers on mobile carriers and shared home IPs do not pay for sharing an address.
• Bot management scoring at the CDN, which combines TLS and HTTP fingerprints with behaviour across many sites.

None of these is perfect, and all of them cost some false positives. Check the error pages and bounce rates on real visitors after each change.

Test the block like a visitor

Three requests show whether the rules do what they claim:

# 1. A declared AI crawler should get 403 on a page...
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://example.com/some-article/
# 2. ...and 200 on robots.txt, so it can read the opt-out
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://example.com/robots.txt
# 3. A normal browser from a home connection should get 200
curl -s -o /dev/null -w "%{http_code}\n" -x http://HOST:PORT -U "USERNAME:PASSWORD" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://example.com/some-article/

The third check matters most for sites that added ASN or IP-reputation rules. Running it from a cloud server tests the wrong network. A home connection in the target market, or rotating residential proxies set to that country, shows what a real reader there gets. Point this test only at sites you run.

Does blocking AI crawlers hurt SEO?

Blocking training crawlers and training tokens has no effect on classic search. Google’s common crawlers page (updated July 14, 2026) says Google-Extended does not affect inclusion in Google Search, and Apple says the same for Applebot-Extended and its search features.

AI Overviews are a different matter. Google’s AI features guide (checked September 2026) treats them as a Search feature: a page needs to be indexed and eligible for a snippet, and Google-Extended does not remove it. The controls that do are Search controls such as nosnippet, data-nosnippet and max-snippet, which also trim the ordinary result snippet.

Blocking AI search crawlers is a real trade. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether the site can appear and be cited in those assistants’ answers. Block them and the citations stop; the crawl-to-click ratios above suggest how many visits that is worth for a given site, and it varies widely by niche.

Should you block AI crawlers?

Whether to block AI crawlers comes down to what the site sells.

• Publishers and paywalled content: block AI crawlers used for training, decide on AI search crawlers per assistant, enforce at the edge and verify. Content is the product.
• E-commerce catalogues: block training crawlers and noisy crawlers by load. Leave search and user fetchers open: a shopper asking an assistant about a product is a lead.
• Docs, SaaS and developer sites: most allow everything, because being the answer inside an assistant is distribution. Rate-limit if crawl load hurts.
• Personal sites: robots.txt plus a Content-Signal line is enough. Add a CDN category block if the logs show heavy traffic.

In every case, keep robots.txt readable, verify the bots you allow, and test from a home connection after any rule that touches IP ranges. When the logs show traffic that no rule explains, those requests can come from four kinds of network: hosting ranges, home connections, ISP-announced static addresses and mobile carriers.