Google, Microsoft, OpenAI and Anthropic Back One AI Training Opt-Out

Google, Microsoft, OpenAI and Anthropic have backed a single robots.txt instruction that would let a website refuse the use of its content for AI training by all participating companies at once. The proposal, announced on Friday 30 October, would replace the current patchwork in which publishers must block each AI company’s crawler separately.

The shared signal is still only a draft and none of the companies says it is honouring it yet, but the backing of four of the biggest AI developers makes it the most significant attempt so far to give website owners a simple, common way to opt out. Meta has said it supports the idea, without committing to follow it.

What Is the New AI Training Opt-Out?

The mechanism is what the draft calls a “policy token”. Instead of naming a specific crawler on the User-Agent line of a robots.txt file, a site names the token __usage_ai-training. A rule such as “User-Agent: __usage_ai-training” followed by “Disallow: /” expresses a preference against AI training for the whole site, and site owners can still allow particular folders, such as a press kit.

The specification was written by Kevin Kelley, a technical lead for standards at Anthropic, and submitted to the Internet Engineering Task Force (IETF) as an individual draft at 15:39 UTC on 30 October, according to PPC Land. It deliberately introduces no new grammar: path matching and precedence follow RFC 9309, the Robots Exclusion Protocol the IETF published in 2022. Until it passes through the IETF process, it is not a formal internet standard.

When Will Google, Microsoft and OpenAI Honour It?

The four companies have given different timetables:

• Microsoft: will honour the new preferences from 1 February 2027, with separate controls for AI training and for “grounding”, in which a model is pointed at a web source to support its answers. Sites will still appear in Bing search results regardless of their choice.
• Google: said in a Search Central blog post by John Mueller that it aims to support the control “in 2027”, and its crawler documentation says “early 2027”. At Google the token will also keep a site out of Gemini grounding.
• OpenAI: said it aims “to implement the signal in the coming months”.
• Anthropic: The Verge reported that the company has already adopted the signal, while Mr Kelley wrote that Anthropic “has already started” implementing support.

Microsoft is also retiring an older method. PPC Land reported that sites relying on NoArchive or NoCache tags to keep content out of Microsoft’s AI will lose that opt-out on 1 March 2027 unless they switch to the new controls.

Does Blocking All Bots Already Count?

No, and this is the detail most site owners are likely to miss. The draft states that wildcard groups, written as “User-Agent: *”, “MUST NOT be interpreted as matching the policy token”. A website that currently blocks every crawler with a single wildcard rule has, under the draft, expressed no preference about AI training at all. To opt out, the token has to be named explicitly.

Existing company-specific tokens such as Google-Extended keep their current meaning, but they will not count as a shared opt-out for other providers. The draft also says private agreements can override the signal, so a publisher with a licensing deal can publish a blanket opt-out while that contract continues to govern its relationship with a particular AI company.

Crucially, the token is about how content is used, not whether it can be fetched. The draft says it “is not a directive for crawling behavior”. Publishers that want a crawler to stop visiting altogether still need to block it by name.

Why Does This Matter for Publishers?

The announcement comes as news organisations, authors and other creators fight over the use of their work to train AI systems, including in copyright lawsuits against Meta and others. Some have turned to commercial tools: the Really Simple Licensing (RSL) standard lets publishers set terms for crawlers, and Cloudflare blocks AI crawlers by default.

“We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Meta’s communications head Andy Stone told The Verge, adding that the company looked forward to working with other AI firms on the standards. He did not say whether Meta would honour it.

For AI companies, a common signal could reduce legal and reputational risk by making it clearer which content they may use. For publishers, it reduces the work of tracking dozens of crawler names. Critics are likely to point out that the signal depends on companies choosing to respect it, and that content already collected is not covered: the draft says the preferences that apply are those in force when the content was retrieved.

What Should Website Owners Do Now?

Nothing changes immediately. Site owners who want to opt out can add the token to their robots.txt files now so it is in place when the companies begin reading it, while keeping existing crawler-specific blocks. Those who want their content to appear in AI answers, which increasingly drive traffic through tools that rely on retrieval-augmented generation, may prefer Microsoft’s and Google’s separate grounding controls once they are live.

The move also follows a busy week for the AI industry, in which CNBC reported that OpenAI had cut the target for its latest funding round to up to $20bn from about $30bn. Anthropic, whose engineer wrote the draft, has also been preparing for a stock market listing.

Featured image: Server racks in a data centre, by Carl Lender / Wikimedia Commons, CC BY 2.0. Illustrative file photo.

Sources

• The Verge — The biggest names in AI are backing a new way to opt out of scraping (30 October 2026).
• PPC Land — Google, Microsoft, OpenAI and Anthropic back one AI training opt-out (30 October 2026).
• OpenAI — Helping websites express AI training preferences (30 October 2026).
• CNBC — OpenAI pares fundraising goal, now seeking up to $20 billion in latest round (31 October 2026).