Google, Microsoft, Anthropic and OpenAI want one shared robots.txt
On October 30, four companies proposed a shared robots.txt signal to accept or reject AI training on a site's content. The promise is simple; the control is anything but.
One shared signal in a familiar file
In an article published on November 6, Next reports that Google, Microsoft, Anthropic and OpenAI want to share a single rule in robots.txt. Their goal is to let a site indicate whether it accepts or refuses the use of its content to train generative AI models.
The text is called Machine-Readable Training Preference Signal. It was written by Kevin Kelley, an engineer at Anthropic, and aims to be standardised by the IETF, the organisation responsible for Internet technical standards. The draft is due to expire on May 3, 2027, if no new version is published by then. Even provisional standards come with a sell-by date: the web may run on cookies, but it prefers its drafts fresh.
One line for several robots
Today, a publisher that wants to refuse training has to target the relevant robots separately. Next mentions GPTBot for OpenAI and Google-Extended for Google, among others. The proposal adds a new token, __usage_ai-training, placed in the User-Agent line.
A site could then express a shared preference using Disallow to refuse training or Allow to authorise it, path by path. The draft would therefore allow a publisher to block the whole site while keeping a press area accessible. The principle is easier to read on the publisher’s side: declare one rule once, provided the companies involved agree to read it.
The four groups announced on October 30 that they would respect this signal, with different support schedules over the coming months. The proposal does not turn robots.txt into a universal lock. It is closer to putting the same sign on a door that crawlers already know.
Refusing does not close the door
First limitation: the new token does not block a site from being crawled. Rules concerning robots continue to apply separately. Content can therefore be collected without this directive deciding, by itself, how it may be used for training.
Second limitation: the signal is declarative and voluntary. It only binds providers that commit to respecting it, and the draft provides neither monitoring nor penalties. A publisher has no way to check whether refused content was used to train a model. The text also applies only to content collected after it is added: it cannot extract anything from models that have already been trained.
A rule with no referee
OpenAI presents the initiative as voluntary, and robots.txt itself has no binding legal force. Companies that do not take part are not covered; Next cites Apple, Amazon and Meta among those that have not committed to it. Nor is there anything requiring other model developers to read the new token, any more than there is anything forcing robots that already ignore robots.txt to start paying attention.
The idea does have one clear advantage: it replaces a pile of separate settings with a signal people can understand. But its effectiveness depends entirely on the recipients’ goodwill, and on whether a publisher can trust them without any way to check. It is a digital handshake: useful for confirming who is in the room, rather less useful if one side decides to keep its hands in its pockets.
Sources (1)
Written with AI assistance from the sources cited above, then reviewed and approved before publication by Sébastien Soulier.
