OpenAI, Google, Microsoft and Anthropic back a single robots.txt token to opt out of AI training
On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor a single robots.txt policy token, __usage_ai-training, to express a site’s refusal of generative AI training in one directive. The signal is voluntary, non-retroactive, and rolling out on four different calendars, so it must be added alongside the existing per-crawler rules.
October 30, 2026. OpenAI, Google, Microsoft and Anthropic announce they will honor a single robots.txt policy token, __usage_ai-training, so a website can express its refusal of generative AI training in one directive. October 30, 2026, same day. Kevin Kelley, standards lead at Anthropic, files the IETF Internet-Draft “Machine-Readable Training Preference Signal” that defines the token. October 30, 2026, again. The four companies do not commit to the same date: Anthropic has already started, Microsoft targets February 1, 2027, and Google some time in 2027. The headline: opting out of training becomes a single, machine-readable preference across four crawlers — but it does not apply to the past, and it does not govern crawling.
For a publisher that has maintained one robots.txt line per crawler — GPTBot, Google-Extended, ClaudeBot — the announcement collapses a chore into a single entry. It does not eliminate it: the token is a voluntary promise, not a ratified standard.
What the token actually does
The mechanism reuses the existing robots.txt grammar without adding new directives. A site writes two lines to refuse everything, with the option of Allow exceptions:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/ A Disallow path expresses a preference against training; an Allow path expresses one in favor. The generic **User-Agent: *** group does not count, and a file that never names the token expresses no preference at all. The definition of “training” is borrowed from the IETF AI Preferences working group: using content to change the learned parameters of a generative model.
The document carrying it is an individual Internet-Draft, filed by Kelley, dated October 30, 2026, and expiring May 3, 2027. The IETF notes it is not endorsed and has no formal standing in its standards process. The AIPREF working group, co-chaired by Mark Nottingham (Cloudflare) and Suresh Krishnan (Cisco), includes Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation, and Creative Commons.
Four commitments, four calendars
The four companies do not honor the token on the same schedule. Anthropic has already started implementing it; OpenAI promises to honor it “in the coming months” while keeping its per-crawler rules; Microsoft targets February 1, 2027 and plans a separate setting for grounding, the retrieval step that pulls live content into answers; Google will treat the token as equivalent to a Google-Extended opt-out, for a 2027 rollout.
For a publisher, that means the token is only partly effective on day one. Until each company ships support, the specific per-crawler rules remain the dependable route. Microsoft also draws a line between two kinds of consent: training changes a model’s weights, while grounding displays or summarizes content at query time — and publishers often treat those differently.
Three limits worth keeping straight
The announcement was welcomed, but it does not turn robots.txt into a security control. Three limits frame what it really changes.
- It does not govern crawling. A crawler may still fetch a page that carries the opt-out. Blocking access still requires conventional crawler rules or technical controls.
- It is voluntary. OpenAI describes it as a “coordination mechanism” under RFC 9309 (the Robots Exclusion Protocol), not a security control. A scraper that ignores robots.txt will ignore the token too.
- It is not retroactive. Updating robots.txt does not remove material already present in datasets or trained models.
The same day, OpenAI also published a compilation of court briefs arguing that training on copyrighted work is fair use in its dispute with The New York Times. Read together, the two announcements draw the big labs’ position: a cheaper, more consistent way for publishers to say no, while they argue in court that training requires no prior permission at all.
What to do now
The practical guidance comes down to two moves, done together rather than one instead of the other.
- Add the token now. Two lines in robots.txt suffice, with Allow rules for the areas you want to open. It is risk-free and ready for the day each lab switches it on.
- Keep the per-crawler rules. GPTBot, Google-Extended, ClaudeBot and the rest remain the dependable route until all four commitments ship through 2027.
- Do not confuse training with crawling. If the goal is to block collection itself, the token is not enough: you need blocking rules or server-side controls.
The real deadline is May 3, 2027, the draft’s expiry date. Until then the document can change, and real-world adoption will show up in server logs, company by company.
Verdict
The announcement turns a scattered chore into a readable preference, but it replaces nothing. If you publish content and refuse AI training, add the __usage_ai-training token now and keep your per-crawler rules: you will be ready without losing anything. If you run a site that must also block collection, treat the token as a preference signal, not a barrier, and maintain your technical controls. The lesson fits in one line: refusing training just became a de facto standard, but it is a voluntary, prospective, and slow-to-materialize one.