Google, OpenAI, Microsoft, and Anthropic are backing a new directive that would allow websites to control how their data is used by AI models. This common signal, intended to complement the robots.txt protocol, aims to provide publishers with a simpler way to object to their content being used to train models.
- Google, OpenAI, Microsoft and Anthropic back a common signal that lets websites control AI training.
- The signal, named __usage_ai-training, would be added to the robots.txt file.
- It separates model training from search indexing and AI-generated responses.
- The rollout begins in 2027, with Microsoft starting on February 1.
Until now, website administrators had to set up separate rules for each of the crawler bots used by tech companies. Google offers Google-Extended, while OpenAI has GPTBot. This proliferation of mechanisms complicates permission management, especially for media outlets and content producers whose publications are likely to be integrated into model training data.
The new proposal relies on a common identifier, named __usage_ai-training, which could be added directly to the robots.txt file. Submitted to the Internet Engineering Task Force (IETF), it was filed by Kevin Kelley, an engineer at Anthropic. It is designed to allow site administrators to define their preferences, applicable to the various AI providers that adopt this mechanism. A site could thus prohibit the use of all its pages for model training, while allowing certain sections, such as its press area.
Separating Model Training and Visibility
One of the challenges of this evolution concerns the distinction between training AI models and indexing content in search engines. Google states that the new signal should allow publishers to refuse the use of their pages for training without necessarily blocking their exploration by search engines. The group specifies that its existing mechanisms, including Google-Extended, will continue to function alongside this new rule.
Microsoft also advocates for this separation. In its presentation of the new preferences for publishers, the company distinguishes model training from the use of content to fuel responses generated by AI systems, a practice often referred to as grounding. The publisher will be able to manage these two uses separately, and Microsoft intends to maintain the ability for sites to remain visible in Bing, regardless of their choices regarding model training.
A Staged Rollout Starting in 2027
The participating companies plan a gradual rollout of the mechanism. Microsoft announces support starting February 1, 2027, as its bots crawl sites. Google also targets integration in 2027, while Anthropic has already begun implementing this directive and OpenAI plans to integrate it in the coming months.
The rules specific to each provider will remain usable in addition to the common signal.
