🔍 Read the full analysis: What The Shared AI Training Opt-Out Covers—and What It Leaves Out on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, will take effect on different schedules, and does not block crawling, remove content from existing models or opt a site out of AI answers based on live web content.
The proposed rule uses a shared user-agent name, __usage_ai-training, followed by Disallow: / in a site’s robots.txt file. It expresses a publisher preference against training use. The mechanism is a voluntary instruction to participating crawlers, not a technical barrier that prevents a company from accessing a page.
The companies have different implementation timelines. Anthropic says it has started implementation, though it expects support across its services to take time. OpenAI says it aims to implement the signal in the coming months. Microsoft will honor it from February 1, 2027, as it crawls websites, while Google says it aims to support the control in 2027 and its crawler documentation specifies early 2027.
Microsoft is also introducing a separate page-level preference for AI Grounding—content used to inform answers in Bing, Copilot and Microsoft 365. It says the new preferences take effect February 1, 2027. Microsoft’s existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. Other company-specific crawler rules, including GPTBot, Google-Extended and ClaudeBot controls, remain in place during the rollout.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, a common instruction could make it easier to communicate a training preference without maintaining a separate rule for every participating company. That matters because the current controls differ by provider, and a publisher that misses a company-specific rule may not communicate its preference to that company.
But the signal does not create an enforceable block or guarantee that a site will be excluded from every AI-related use. Training, AI answers and search are distinct choices: Microsoft’s separate grounding setting makes that distinction explicit. Microsoft says its preferences do not change eligibility for Bing search results; Google says Google-Extended does not affect Google Search. The announcement also does not offer publishers payment for content. Microsoft says neither preference guarantees inclusion, traffic or payment.
For readers, the main consequence is that publishers gain a simpler way to state a preference, not a guarantee about how every AI service will use online material. The signal also concerns website content, not an individual’s conversations or account settings in an AI assistant.
Products Worth Considering
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
As an affiliate, we earn on qualifying purchases.
Company Rules Remain in Place
The shared signal is described in an individual Internet-Draft titled Machine-Readable Training Preference Signal, written by Kevin Kelley of Anthropic and published October 30. It builds on work by the Internet Engineering Task Force’s AI Preferences working group, but the draft is not an official internet standard. At a working-group meeting on October 21, a chair said the group had not declared consensus on anything.
Company statements point to the same draft but differ in scope and timing. Anthropic’s announcement says implementation has begun, while its crawler help page still described existing per-crawler controls on October 31. Google says that, if a site uses the new token and has no Google-Extended rule, it plans to interpret the shared preference as though the publisher had disallowed Google-Extended. OpenAI describes its commitment as a voluntary effort concerning foundational generative AI training.
Microsoft’s announcement goes beyond the shared training signal by specifying a separate grounding preference and a schedule for replacing the role of NoArchive and NoCache. Publishers using those tags will need to review their settings before the March 2027 cutoff if they want them to communicate training or grounding preferences.
““strongly support[s] publishers’ ability to express preferences””
— Meta spokesperson Andy Stone
Adoption and Enforcement Remain Open
Whether companies will comply in practice cannot be guaranteed by the file itself. Robots.txt communicates a preference; it does not prevent a crawler from retrieving content. The draft also has no formal standing in the IETF standards process, and the working group has not declared consensus on it.
Meta has expressed support for publishers’ ability to state preferences but has not said whether it will honor this particular signal. The source material reviewed also found no commitment from Apple, Amazon, xAI or Perplexity. The list of participating companies may change, but support beyond the four named companies is unconfirmed.
It is also not yet clear exactly how every company will apply the rule across its services, datasets and existing systems. The signal is aimed at future training use; it does not undo training already completed. Nor does it cover live-web retrieval or grounding unless a separate control addresses those uses. Details may become clearer as companies publish technical documentation and roll out support.
Watch the 2027 Rollout
Website operators can add the shared training rule if they want participating companies to receive that preference, while retaining existing company-specific directives until each provider’s support is active. Publishers should check individual crawler documentation as dates and implementation details develop rather than assuming one line takes effect everywhere immediately.
The next stated milestones are OpenAI’s planned implementation in the coming months, Google’s target of early 2027, and Microsoft’s February 1, 2027 start date. Microsoft says NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027. Anthropic’s rollout across its services remains in progress. The IETF working group’s future discussion will determine whether the draft advances through a standards process; no adoption date has been announced.
Key Questions
What does the shared AI training opt-out do?
It adds a robots.txt preference against using a website’s content to train generative AI models. It is intended for companies that choose to honor the signal.
Does it stop AI companies from crawling a site?
No. The signal is a request, not a technical block on access. It also does not remove content from models that have already been trained.
Not by itself. The shared rule concerns training. AI answers and grounding are separate uses; Microsoft announced a separate AI Grounding preference for its services.
Which companies have said they will honor it?
OpenAI, Google, Microsoft and Anthropic have announced support, with different implementation timelines. Meta expressed support for publishers stating preferences but did not commit to honoring this signal.
Is this an official internet standard?
No. It is an individual Internet-Draft associated with the IETF’s AI Preferences working group. The draft has no formal standing in the standards process, and the group has not declared consensus.
Source: ThorstenMeyerAI.com
