OpenAI and Other Companies Announce Support for Shared robots.txt Training Opt-Out Standard
Understand the story
On October 30, 2026, OpenAI announced it would honor a unified robots.txt signal. This mechanism is designed to help website operators express their preferences on AI training to AI providers: a website only needs to declare its training preference once via robots.txt to communicate it to OpenAI and other participating providers, without having to configure each provider separately. On October 31, according to The Verge, OpenAI, Google, Microsoft, and Anthropic announced they would jointly adopt a new shared standard that lets publishers use a policy token in robots.txt to opt out of AI training crawling in one step across multiple AI companies. Compared with OpenAI's earlier standalone announcement, this development clarifies the list of other major providers participating in the mechanism. The reports have not yet disclosed details such as the specific implementation timeline of the standard.
AI overview of reporting · Updated 2 h前
Reporting timeline
Explore different aspects of the story through its reporting.
- The Verge · AIThe biggest names in AI are backing a new way to opt out of scraping
OpenAI, Google, Microsoft, and Anthropic announced they will adopt a new shared standard that lets publishers opt out of AI training scraping with multiple AI companies at once via a policy token in their website's robots.txt.
- OpenAI NewsMaking it easier for websites to express AI training preferences
OpenAI announced it will respect unified robots.txt signals, helping website operators communicate their AI training preferences to participating providers. Sites can state their preferences once via robots.txt to notify OpenAI and other participating providers, without configuring each one separately.
Story heat trend
Current heat17·Comparable peak17（10/31 06:00）·Comparable change over 24 hours –
The trend compares the same continuously observed sources, which may cover fewer sources than the current heat score. Hover or select the chart to inspect each hour; use the left and right arrow keys to navigate.