robots.txt AI training opt out: one token, four companies
One robots.txt AI training token asks OpenAI, Google, Microsoft and Anthropic not to train on your site. How to add it, when each honours it, how to test it.
Quick summary · TL;DR
- Add two lines,
User-agent: __usage_ai-trainingandDisallow: /, and the opt-out is in place. OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 they'll honour that group, which saves a separate rule for every training bot. - Don't expect it to work everywhere yet. Anthropic says it has begun recognising the signal. Microsoft starts on 1 February 2027, while OpenAI talks about the coming months and Google aims for 2027.
- Check your GPTBot, ClaudeBot and Google-Extended lines against the token, because they still count and can overrule it. A rule that names ClaudeBot wins at Anthropic and an explicit Google-Extended rule wins at Google, so the two have to agree.
- Put Allow lines first, since the draft's own example trips Python's parser. Given
Disallow: /aboveAllow: /press/, Python 3.9.6's robotparser refused /press/kit too.
The robots.txt AI training opt out now fits in two lines, User-agent: __usage_ai-training and Disallow: /, and that one group asks AI companies not to train on your site. OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 that they will honour it, each on its own timeline. It asks rather than blocks, and some parsers care about the order of your Allow lines.
The robots.txt AI training opt out
There’s no new syntax to learn. It’s an ordinary robots.txt group whose User-agent line carries a policy token where a crawler name would sit.
User-agent: __usage_ai-training
Disallow: /Disallow under it states a preference against AI training, and Allow states one in favour. Behind the signal is draft-kelley-aipref-training-preference-signal-00 (30 October 2026), written by Kevin Kelley of Anthropic. It’s an individual Internet-Draft with no formal standing, and it expires on 3 May 2027 unless renewed.
Three rules in the draft are easy to miss.
- A
User-agent: *group never counts for the token. - No
__usage_ai-trainingline in the file means you’ve expressed no preference at all. - The token isn’t a crawl directive, so crawling is still set per crawler name.
Who honours it, and when
| Company | Status, per its own page | When a named rule disagrees |
|---|---|---|
| Anthropic | Began recognising the signal as of 30 October 2026, rolling out over time | A rule that names ClaudeBot wins |
| OpenAI | Aims to implement it “in the coming months” | Existing GPTBot controls keep applying |
| Microsoft | Honours it from 1 February 2027 as Bing crawls | Page-level <meta name="Bingbot" content="ai-training:disallow"> or X-Robots-Tag beats robots.txt |
| Aims for 2027; its crawler docs say early 2027 | An explicit Google-Extended rule wins |
Sources: Anthropic’s crawler article (updated November 2026), then OpenAI’s announcement, Bing’s post and Google’s Search Central post, all three from 30 October 2026.
Per-bot lines and Content-Usage
Your GPTBot, ClaudeBot and Google-Extended groups don’t go away, and the draft allows companies to treat their own named rules as more specific. So a file that disallows the token but allows Google-Extended has told Google yes.
Content-Usage is a separate mechanism from the IETF AI Preferences working group. Its attachment draft (14 September 2026) puts Content-Usage: train-ai=n inside an existing group or in an HTTP header, while the token gets a group of its own.
Check how your parser orders the rules
The draft’s own example puts Disallow: / first and then Allow: /press-kit/. RFC 9309 (September 2022, checked 7 November 2026), section 2.2.2, says the most specific match must be used, meaning the rule with the most octets. So /press-kit/ should come out allowed.
On 7 November 2026 proxymint ran that shape through Python 3.9.6’s standard urllib.robotparser:
User-agent: __usage_ai-training
Disallow: /
Allow: /press/
User-agent: *
Allow: /__usage_ai-trainingon /blog/post: False. On /press/kit: also False, so the Allow was ignored.GPTBot,ClaudeBotandMozilla/5.0 (compatible; ClaudeBot/1.0): True on both paths, because they fall through to the*group.- With
Allow: /press/moved aboveDisallow: /: /blog/post False, /press/kit True.
That parser takes the first matching rule in file order. Only 3.9.6 was tested here.
Run the check yourself
Standard library only:
import re
import sys
import urllib.request
import urllib.robotparser
TOKEN = "__usage_ai-training"
site = sys.argv[1].rstrip("/")
text = urllib.request.urlopen(site + "/robots.txt").read().decode("utf-8", "replace")
# A stock parser falls back to * for unknown names, and the draft says
# * never speaks for the token, so look for the group first.
if not re.search(r"(?im)^\s*user-agent:\s*" + re.escape(TOKEN) + r"\s*$", text):
sys.exit("No __usage_ai-training group: no preference stated.")
rp = urllib.robotparser.RobotFileParser()
rp.parse(text.splitlines())
for path in sys.argv[2:] or ["/"]:
print(path, "allowed" if rp.can_fetch(TOKEN, site + path) else "refused")Run it as python3 check.py https://example.com / /press/. When a path you meant to allow prints “refused”, reorder the group.
What to put in robots.txt now
It depends on how far out of AI you want to be.
- No AI training anywhere. Add the token group, and check that none of your GPTBot, ClaudeBot or Google-Extended lines allows what it refuses.
- In AI search answers but out of training. Use the token and leave search crawlers such as OAI-SearchBot alone.
- Out of Microsoft’s AI answers as well. Microsoft sets grounding per page only, with
<meta name="Bingbot" content="ai-grounding:disallow">orX-Robots-Tag: Bingbot: ai-grounding:disallow. If NoArchive or NoCache was doing that job, switch by 1 March 2027, when Bing stops reading them as a training or grounding preference.
Google needs one more look. Where you have no Google-Extended group, it reads the token as a Google-Extended disallow. That one Google-Extended rule covers both Gemini training and grounding in Gemini Apps, so an explicit Google-Extended allow brings back both.
Crawler builders sit on the other side of the robots.txt AI training token, and the test above shows a stock parser won’t look it up for them. The guide to proxies for AI training data has a gate that reads per-bot tokens and Content-Usage, and the token is one more lookup. A crawl through datacenter proxies or any other exit gets the same file from every IP, so fetch robots.txt once per host, not once per exit, and refresh it within 24 hours, the cache limit in RFC 9309 section 2.4. That refresh is how soon a new opt-out reaches you. Web Bot Auth covers agents that sign their requests.
Frequently asked questions
Add a group with __usage_ai-training on its User-agent line and Disallow: / under it, which OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 they will honour. Any GPTBot, ClaudeBot or Google-Extended groups already in the file should stay consistent with it.
It's a robots.txt token that goes where a crawler name would, in the User-agent line of a normal group, and Disallow paths under it say no to AI training while Allow paths say yes. It comes from draft-kelley-aipref-training-preference-signal-00, an individual Internet-Draft by Kevin Kelley of Anthropic dated 30 October 2026. So it has no formal IETF standing yet.
No, it only speaks to how content is used. The draft says the token is not a crawl directive, and Microsoft puts it as preferences that govern use of content, not crawl access. Stopping a crawler from fetching pages still takes a group under its own name, and even robots.txt itself is a request rather than access control.
Keep them, because they still work and can take priority over the token. At Anthropic a rule that names ClaudeBot beats the general signal and Google follows an explicit Google-Extended rule. OpenAI says its existing crawler controls still apply. Any company that adopts the token later is covered by it too, so the safe setup is both, saying the same thing.
Microsoft says from 1 February 2027 as Bing crawls, while OpenAI and Google give targets. OpenAI aims to implement it in the coming months. Google aims for 2027, and its crawler documentation says early 2027. Anthropic is ahead of all three, since it says it began recognising the signal as of 30 October 2026 and is rolling it out over time.
Not according to the announcements, which look forward. Microsoft describes the preference as covering future model training. None of the four companies says the signal removes data already collected, and it applies as each company reads your robots.txt again.
They sit in different places, since the __usage_ai-training token is a group of its own and Content-Usage is a line such as Content-Usage: train-ai=n inside an existing robots.txt group, or an HTTP header. Content-Usage comes from the IETF AI Preferences working group's attachment draft. Older parsers ignore the token group like any unknown bot, and a site can use both.
Researched and drafted with AI tools, then checked against our publishing rules before it went live: 7 outside links, 3 links to other proxymint pages, no prices and a fact-check pass against our product records. How we write.