--- title: "Google, Microsoft, Anthropic and OpenAI want one shared robots.txt" description: "On October 30, four companies proposed a shared robots.txt signal to accept or reject AI training on a site's content. The promise is simple; the control is anything but." canonical: https://inyourgeek.com/en/articles/2026-11-08-google-microsoft-anthropic-et-openai-veulent-un-robots-txt type: article author: "Sébastien Soulier" publisher: "InYourGeek" language: en datePublished: 2026-11-08 dateModified: 2026-11-08 section: "AI" tags: ["ia", "robots", "entrainement", "openai"] --- # Google, Microsoft, Anthropic and OpenAI want one shared robots.txt > On October 30, four companies proposed a shared robots.txt signal to accept or reject AI training on a site's content. The promise is simple; the control is anything but. November 8, 2026 · AI · 3 min ## One shared signal in a familiar file In an article published on November 6, [Next reports](https://next.ink/266259/google-microsoft-anthropic-et-openai-veulent-changer-robots-txt-pour-refuser-les-ia/) that Google, Microsoft, Anthropic and OpenAI want to share a single rule in `robots.txt`. Their goal is to let a site indicate whether it accepts or refuses the use of its content to train generative AI models. The text is called `Machine-Readable Training Preference Signal`. It was written by Kevin Kelley, an engineer at Anthropic, and aims to be standardised by the IETF, the organisation responsible for Internet technical standards. The draft is due to expire on May 3, 2027, if no new version is published by then. Even provisional standards come with a sell-by date: the web may run on cookies, but it prefers its drafts fresh. ## One line for several robots Today, a publisher that wants to refuse training has to target the relevant robots separately. Next mentions `GPTBot` for OpenAI and `Google-Extended` for Google, among others. The proposal adds a new token, `__usage_ai-training`, placed in the `User-Agent` line. A site could then express a shared preference using `Disallow` to refuse training or `Allow` to authorise it, path by path. The draft would therefore allow a publisher to block the whole site while keeping a press area accessible. The principle is easier to read on the publisher’s side: declare one rule once, provided the companies involved agree to read it. The four groups announced on October 30 that they would respect this signal, with different support schedules over the coming months. The proposal does not turn `robots.txt` into a universal lock. It is closer to putting the same sign on a door that crawlers already know. ## Refusing does not close the door First limitation: the new token does not block a site from being crawled. Rules concerning robots continue to apply separately. Content can therefore be collected without this directive deciding, by itself, how it may be used for training. Second limitation: the signal is declarative and voluntary. It only binds providers that commit to respecting it, and the draft provides neither monitoring nor penalties. A publisher has no way to check whether refused content was used to train a model. The text also applies only to content collected after it is added: it cannot extract anything from models that have already been trained. ## A rule with no referee OpenAI presents the initiative as voluntary, and `robots.txt` itself has no binding legal force. Companies that do not take part are not covered; Next cites Apple, Amazon and Meta among those that have not committed to it. Nor is there anything requiring other model developers to read the new token, any more than there is anything forcing robots that already ignore `robots.txt` to start paying attention. The idea does have one clear advantage: it replaces a pile of separate settings with a signal people can understand. But its effectiveness depends entirely on the recipients’ goodwill, and on whether a publisher can trust them without any way to check. It is a digital handshake: useful for confirming who is in the room, rather less useful if one side decides to keep its hands in its pockets. ## Sources - [Google, Microsoft, Anthropic et OpenAI veulent changer robots.txt pour refuser les IA](https://next.ink/266259/google-microsoft-anthropic-et-openai-veulent-changer-robots-txt-pour-refuser-les-ia/) — next.ink _Article written with AI assistance from the cited sources, reviewed and approved by a human before publication._ --- Equivalent HTML page : https://inyourgeek.com/en/articles/2026-11-08-google-microsoft-anthropic-et-openai-veulent-un-robots-txt Published by InYourGeek — https://inyourgeek.com/en/a-propos Method and safeguards : https://inyourgeek.com/en/methode All articles : https://inyourgeek.com/en/archives