Four AI Companies Back Shared Opt-out For Training Crawlers
Google, OpenAI, Anthropic and Microsoft say they will honor a common robots.txt signal that lets publishers set AI training preferences across providers.
Google, OpenAI, Anthropic and Microsoft say they will honor a shared signal that lets website operators opt out of having their content scraped for generative AI training. The proposed control sits in a site’s robots.txt file, giving publishers a way to state their preferences across multiple companies rather than writing separate rules for each provider.
Table of Contents
How the shared signal works
The proposal was written by Kevin Kelley, Anthropic’s technical lead for standards, and is under consideration at the Internet Engineering Task Force, according to Engadget. The Verge describes it as a shared “policy token” within robots.txt, the file websites use to communicate instructions to crawlers.
Engadget reports that operators can add an AI-training user-agent entry and use Disallow rules to block training crawlers. They can also use Allow rules to permit access to selected parts of a site, such as a press kit. The control is intended to govern scraping for AI training, not to stop all crawling or remove pages from search results. Engadget says Google will apply the signal to Google-Extended, its user agent for gathering material used to train Gemini models.
The directive is forward-looking: it cannot remove content that companies have already collected, Engadget notes.
When companies plan to adopt it
Anthropic already supports the signal. Microsoft says it will begin supporting it on February 1, 2027, while Google plans support in early 2027, according to Engadget. OpenAI says it will implement the control in the coming months. Those timelines mean the four companies’ commitments will not all take effect at once.
The Verge reports that Microsoft also plans more granular publisher controls for AI training and AI grounding, which it describes as pointing a model to a verified information source to reduce hallucinations. Microsoft says a site’s preferences under those controls will not keep it out of Bing search results. The Verge says Meta communications head Andy Stone expressed support for the shared setting, but did not say whether Meta would honor it.
Why publishers are seeking common controls
Today, website operators must identify AI companies’ crawlers and set rules for them individually, according to the reports. A signal recognized by several providers could simplify that work, while leaving publishers able to make distinctions about which parts of their sites may be used for training.
The proposal arrives as publishers contend with AI crawlers collecting site content, The Verge reports. It is one response to that concern, alongside other approaches the outlet describes, including a licensing standard intended to let publishers charge crawlers and Cloudflare’s decision to block AI crawlers by default.
Sources
This story was compiled by AI from the reports below. Read the originals for the full details.