Google, Microsoft and OpenAI have announced they will consider a common line in the robots.txt file to express a preference regarding AI training. A site could thus indicate a single preference to all participating providers, instead of writing separate rules for each company’s bot. None of the three companies currently support this line, according to their respective timelines.
The text is based on an individual draft (‘Internet Draft’) submitted by Kevin Kelley of Anthropic to the IETF’s AI Preferences working group. This document is listed in the IETF’s tracker as an individual draft, which does not represent an official standardization activity of the organization. In a message sent to the group’s mailing list, Kelley states that Anthropic has begun developing this feature.
A standard rule, a specific token
The draft does not add any new directives to the robots.txt file. A rule is written as usual, with Allow and Disallow instructions, under a token called __usage_ai-training. The example given in the text corresponds to a site that refuses training on all its pages, except for the /press-kit/ folder.
The token uses the definition from the working group’s vocabulary draft. Training there refers to the use of content to modify the learned parameters of a generative AI model. Models that only perform non-generative tasks, such as classification or ranking, are excluded. Rules already present in robots.txt files are not considered in this shared preference, and the token does not control page crawling.
Three different approaches
Google indicated, in a post signed by John Mueller, that if a site uses the common line without specifying Google-Extended, the company will interpret it ‘as if the publisher had disallowed Google-Extended’. This control will also apply to content-based responses in Gemini Apps. Google aims for support in 2027, its robots documentation mentioning early 2027. According to this documentation, Google-Extended governs the use of content for training future Gemini models, as well as their use for grounding in Gemini Apps and on Vertex AI. Google does not link this tool to indexing in its search, and AI Overviews and AI mode have separate controls.
Microsoft adopts the common line for training, but adds a separate preference, ‘AI Grounding’, defined only at the page level. According to the post by Krishna Madhavan, Principal Product Manager at Bing, the DisallowExtended option allows grounding in Microsoft tools like Copilot, but not in services provided to other companies, like Web IQ. The company specifies that these restrictions may reduce brand visibility in AI responses, with no effect on ranking in Bing results. Microsoft replaces NoArchive and NoCache with these two new options starting February 1, 2027, and will no longer recognize the old tags after March 1, 2027.
OpenAI and Anthropic
OpenAI aims to support this signal in the coming months, but its current robots.txt policy will remain in effect. Neither its post nor its robots documentation specifies what would happen in case of a conflict between the common line and a GPTBot rule.
As of October 30, Anthropic had not published any announcement on this matter. According to Kevin Kelley’s message, implementation across all the company’s services will take time. Its robots help page only mentions three control options: ClaudeBot, Claude-User and Claude-SearchBot.
Kelley’s draft expires on May 3, 2027, unless replaced by a new version. Google and OpenAI stated they would provide more details as their support is rolled out.
