__usage_ai-training Explained: The Shared robots.txt AI Training Opt-Out Token and Its GEO Impact
OpenAI, Google, Microsoft, and Anthropic have agreed to honor a shared robots.txt token, __usage_ai-training. Here is the syntax, when each company starts applying it, how blocking training affects AI search citations, and the order to change your settings.
What does one __usage_ai-training line in robots.txt actually block, and what happens to your AI search visibility? Site owners started asking on October 30, 2026, when OpenAI, Google, Microsoft, and Anthropic said they would support a shared robots.txt token for refusing AI training (AI Weekly, 2026).[5] Until now, you had to list each company's crawler name separately, GPTBot for one, Google-Extended for another. The short answer: the token covers training only. It does not touch search indexing or AI answer grounding. Set it up correctly and you can refuse training while keeping your path to AI answer citations. Lean on a wildcard group, or confuse it with crawler blocking, and you get something other than what you intended. This page covers what the token means, when each company applies it, the GEO impact, and which lines to change in your robots.txt today.
Definitions in 30 seconds
__usage_ai-trainingis a usage token placed on the User-Agent line of robots.txt that states a preference about using content at a given path for AI training (IETF draft, 2026).[1]- AI training is using content to change the trained parameters of a generative AI model, as defined in the AIPREF vocabulary draft (IETF draft, 2026).[1]
- Machine-Readable Training Preference Signal is the IETF individual draft that defines this token. Anthropic's Kevin Kelley submitted it on October 30, 2026, and it expires on May 3, 2027 (IETF, 2026).[1]
- Google-Extended is a robots.txt control token that decides whether content Google crawls can be used for Gemini training and grounding. It has no effect on Google Search visibility or rankings (Google, 2026).[2]
- Grounding is the step where a model pulls content from a search index at query time to build its answer. It is controlled separately from training (Google, 2026; Search Engine Roundtable, 2026).[2][4]
__usage_ai-training syntax: one group does the job
The draft adds no new directive and no new syntax. You write the usage token where a crawler name would normally go on the User-Agent line of an ordinary robots.txt group. Disallow and Allow path rules, including longest-match precedence, work exactly as RFC 9309 defines them (IETF draft, 2026).[1] Here is the draft's own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Those three lines state a training refusal for the whole site, with /press-kit/ left open for training (IETF draft, 2026).[1] Implementations tend to go wrong in three places:
- The token must match character for character, ignoring only case. Drop one of the two leading underscores and it becomes a different name (IETF draft, 2026).[1]
- A
User-agent: *group does not match this token. Even if your wildcard group already has Disallow, you still need a separate token group (IETF draft, 2026).[1] - A single group can list several tokens and can mix them with company crawler names. Each crawler keeps applying the rules for its own name under RFC 9309 (IETF draft, 2026).[1]
The preference is judged against the robots.txt in effect when the content was fetched (IETF draft, 2026).[1] Adding the token today will not pull content that has already been collected into existing datasets or trained models (Martin Cid Magazine, 2026).[6]
Support timelines and separate controls by company
All four companies have committed to support, but their timing and existing controls differ. The table reflects public sources checked as of November 2, 2026.
| Company | Token support status | When it applies | Existing training control | Search and grounding control |
|---|---|---|---|---|
| Anthropic | Implementation started | Already underway at the October 2026 announcement, rolling out across services in stages | Runs alongside its own crawler name rules | Not addressed separately in these public sources |
| OpenAI | Support planned | In the coming months | GPTBot | OAI-SearchBot, independent of training settings |
| Support planned | Early 2027 | Google-Extended | Googlebot search indexing; Google-Extended does not affect Search visibility | |
| Microsoft | Support planned | Content crawled from February 1, 2027 | AI Training setting | AI Grounding and Limited Grounding settings |
Anthropic began implementing at the time of the announcement and said full rollout across its services would take time (Martin Cid Magazine, 2026).[6] OpenAI committed only to "the coming months" and described the signal as a coordination mechanism, not a security control (AI Weekly, 2026).[5] Its documentation says OAI-SearchBot and GPTBot settings are independent of each other, and gives an example that allows OAI-SearchBot while blocking GPTBot (OpenAI, 2026).[3] Google's crawler documentation states it plans to support the token in early 2027 (Google, 2026).[2] Google will treat a site that uses only the token, with no Google-Extended rule, as blocking Google-Extended. That applies to Gemini training only, and Google Search visibility is unaffected (Martin Cid Magazine, 2026).[6]
Microsoft has the most granular setup. Bing offers three settings, AI Training, AI Grounding, and Limited Grounding. Limited Grounding lets Microsoft's own AI products use your content while excluding third-party grounding services (Search Engine Roundtable, 2026).[4] The schedule runs in this order: interim grounding settings in Bing Webmaster Tools on December 15, 2026; new preferences applied to content crawled from February 1, 2027; a March 1, 2027 deadline for existing NoArchive and NoCache users to switch over; and expiry of the interim grounding settings on September 30, 2027 (Search Engine Roundtable, 2026).[4] According to the report, existing NoArchive and NoCache users who do not move to the new settings by March 1 may not keep their current restrictions (Search Engine Roundtable, 2026).[4]
One caveat. Some of the Microsoft dates come from secondary reporting (Google's timing is confirmed in its crawler documentation), and because each company implements on its own schedule, observers expect uneven compliance for most of 2027 (DEV Community, 2026).[7] Meta has not said whether it will comply (Martin Cid Magazine, 2026).[6]
How blocking training affects GEO and AI answer citations
For GEO, the answer is that the training token and citations are separate switches. AI answers link to your page as a source through search indexing and grounding, and those paths are governed by OAI-SearchBot, Googlebot, and Bing's AI Grounding setting (OpenAI, 2026; Google, 2026; Search Engine Roundtable, 2026).[3][2][4] The token does not block fetching either. The draft defines it as a content usage preference, not a crawl directive (IETF draft, 2026).[1] Bots can still fetch the page; the only thing passed along is a request not to train on what they fetched.
| robots.txt setting | AI training use | AI answer citations (search, grounding) | GEO verdict |
|---|---|---|---|
| No rules | Allowed | Kept | Training and citations both open |
Add only Disallow under a __usage_ai-training group | Refusal stated from each supporter's implementation date | Kept | Default choice for sites that want to refuse training only |
Disallow under User-agent: * | Does not match the token; handled only through company crawler rules | Cut off, because search crawlers without their own named group are blocked too | The setting that loses the most citations |
| Token block plus Bing AI Grounding refusal | Refused | Reduced in Bing and Copilot answers | Gives up visibility in Microsoft answers |
| Token block plus OAI-SearchBot block | Refused | Dropped from ChatGPT search answers | Gives up visibility beyond navigational links |
Microsoft has said that refusing grounding can sharply reduce a site's visibility in Bing, Copilot, and Microsoft 365 answers (Martin Cid Magazine, 2026).[6] OpenAI's documentation likewise says sites that block OAI-SearchBot will not appear in ChatGPT search answers (OpenAI, 2026).[3] Whether to refuse training is a business call. If you want to keep your citations, though, do not implement a training refusal by blocking search crawlers. The same mix-up showed up as the mixed-purpose crawler problem in Cloudflare's default AI crawler block.
The limits are clear. robots.txt is not a security control that restricts access; it only binds crawlers that choose to follow it (RFC Editor, 2022).[8] The source document is an individual draft with no formal standing in the IETF standards process (IETF, 2026).[1] To become a formal standard, it would have to pass through the IETF AI Preferences working group (Martin Cid Magazine, 2026).[6]
Action plan: a five-step robots.txt check
| Step | What to do | How to check |
|---|---|---|
| 1. Inventory current rules | List every existing group, including User-agent: *, GPTBot, and Google-Extended | Review the raw robots.txt alongside your CDN bot settings |
| 2. Decide your goals | Decide separately whether to refuse training and whether to keep search citations | Build a table with separate training and citation columns for each company |
| 3. Add the token group | To refuse training, put Disallow under a __usage_ai-training group and leave shareable paths under Allow | Check the token spelling and both leading underscores |
| 4. Keep existing rules | Leave GPTBot and Google-Extended rules in place until each support date has passed. Bing users should move to the new settings before March 1, 2027 | AI settings screen in Bing Webmaster Tools |
| 5. Track citations | Compare citation frequency and link changes in AI answers, engine by engine, before and after the change | Search crawler requests in server logs, plus a citation tracking tool |
Server logs alone will not cover step 5. You have to check repeatedly how often each engine's answers cite your domain before you can tell whether a training setting change affected citations. Teams use global tools such as Profound and Otterly.ai for this, or Korean tools such as BOIDA, which tracks Korean-language answers. For crawler-by-crawler access policy, see the AI crawler management guide. For options specific to Google's AI Mode, see the Google AI Mode publisher opt-out strategy.
Wrap-up
__usage_ai-training consolidates the training refusal signals that used to differ by company into a single robots.txt group (IETF draft, 2026).[1] It covers training only and leaves search indexing and grounding alone, so used correctly it lets you refuse training without losing AI answer citations. Each company applies it on its own timeline through 2027, and it does not reach back into data that has already been trained on. The job right now is to add the token group without deleting your existing crawler rules, and to confirm that a wildcard block has not shut your citation path as well. For GEO fundamentals, start with What is GEO; for the market as a whole, see the 2026 global GEO and AEO landscape. For how this relates to site structure signals, read the complete llms.txt guide together with Korean blog platform AI crawler policies.
Related companies
- 보이다 (BOIDA)생성형 검색 최적화(GEO) 솔루션, AI 가시성 측정
- Otterly.aiAI 가시성 모니터링 툴
- ProfoundAI 가시성 모니터링 플랫폼
Frequently asked questions
- No. The token only states a preference about AI training. OpenAI handles search visibility through OAI-SearchBot, Google handles it through Googlebot indexing, and Microsoft has split grounding into its own setting. Leave your search and grounding rules as they are and your answer citation path stays intact.
- No. The draft says wildcard groups must not be matched to this token. To state a training preference, create a separate group with __usage_ai-training written by name on the User-Agent line.
- No. The source document is an IETF individual draft with no formal status in the IETF standards process. Compliance rests on each company's voluntary commitment, and robots.txt itself is not a security control that blocks access. Becoming a formal standard would require going through the IETF AI Preferences working group.
- No. The draft judges the preference by the robots.txt in effect when the content was fetched, and press coverage notes that nothing is pulled from existing training data or models already trained on it. Each company applies the token only to content collected after its own implementation date.
- Keep them for now. The draft states that company-specific crawler rules continue to apply, and implementation dates differ by company through 2027. Google says a site that uses only this token, without a Google-Extended rule, will be treated as blocking Google-Extended, but that support arrives in early 2027.
- As of late October 2026, Meta had not said whether it will comply. Meta only called a shared signal an important step forward. Keep writing individual rules for Meta's crawlers the way you do today.
Q.If I add __usage_ai-training to robots.txt, will my site disappear from ChatGPT or Google's AI answers?
Q.My robots.txt already has Disallow under User-agent: *. Does that cover this token?
Q.Is this token a legally binding standard?
Q.Will content that was already used for training be removed?
Q.Can I delete my GPTBot or Google-Extended rules?
Q.Does Meta honor this token?
Sources
- [1] ↑Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) — IETF Datatracker
- [2] ↑Google's common crawlers — Google Search Central
- [3] ↑Overview of OpenAI Crawlers — OpenAI
- [4] ↑Microsoft Bing New AI Controls Separate AI Training From AI Search Grounding — Search Engine Roundtable
- [5] ↑OpenAI, Google, Microsoft, Anthropic back one AI opt-out token — AI Weekly
- [6] ↑Google, OpenAI, Anthropic and Microsoft agree on a robots.txt AI training opt-out — Martin Cid Magazine
- [7] ↑OpenAI, Google, Microsoft and Anthropic to support a unified AI training opt-out token in robots.txt — DEV Community
- [8] ↑RFC 9309: Robots Exclusion Protocol — RFC Editor
Related documents
- Managing AI Crawlers: Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offsHow to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt: based on OpenAI's and Google's official documentation.
- Cloudflare's Default AI Crawler Block, September 15, 2026: Will Your Site Vanish from AI Answers?A breakdown of the Search, Agent, and Training defaults Cloudflare put in force on September 15, 2026. Search crawlers stay allowed, while Agent blocks and mixed-purpose crawler inheritance are the real paths that cut citations in AI answers.
- Google AI Mode and AI Overviews Publisher Opt-Out: The GEO Strategy Dilemma and Decision FrameworkGoogle rolled out AI Mode and AI Overviews opt-out toggles in Search Console in June 2026. Opting out eliminates GEO visibility entirely; staying in means AI consumes content for free while traffic erodes. Here is how the dilemma is structured and how different publisher types should choose.
- Korean Blog Platform AI Crawler Policies Compared 2026: Naver Blog, Tistory, and Brunch robots.txtNaver Blog blocks GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot outright in robots.txt. Tistory and velog carry no AI-crawler-specific rules, and Brunch blocks training bots only. Here is what each platform's raw robots.txt says and how that changes an article's odds of being cited in AI search.
- The Complete llms.txt Guide, Authoring, Operations, and Whether It WorksWhat llms.txt and llms-full.txt are, why they were proposed, and how to author and maintain them, with worked examples. Covers how they differ from robots.txt, the debate over whether they actually work, and a practical checklist.
- What Is GEO: The Definition of Generative Engine Optimization and How It Differs From SEOGEO (Generative Engine Optimization) is the strategy of getting your content cited in answers produced by generative engines like ChatGPT and Perplexity. Here is the definition, how it differs from SEO, and how it works.
- Global GEO/AEO Player Landscape 2026, Monitoring Tools, Agencies, and PlatformsA 2026 landscape that sorts GEO/AEO players into monitoring tools, specialist solutions and agencies, enterprise platforms, and regional players. We compare the leading vendor in each category, founding, headquarters, tracked engines, pricing, and differentiation, against primary sources.