WWikiAP
Category: Technical

Types of AI Crawlers: Search vs. Training vs. User-Triggered Agents, and How robots.txt Applies

GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, ChatGPT-User and four more AI bots sorted into training, search indexing, and user-triggered fetchers, with robots.txt behavior taken from OpenAI, Anthropic, and Google documentation.

Technical GEO 에디터Published

Write robots.txt rules without knowing which kind of AI crawler you're dealing with, and you can get the opposite of what you wanted. A site that only meant to opt out of training vanishes from ChatGPT search answers. A site that thought it had blocked everything keeps getting fetched whenever a user asks about it. Both mistakes have one cause. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and Google-Extended have similar names, but they do different jobs and robots.txt covers them differently. This guide sorts those nine names into training crawlers, search indexing crawlers, and user-triggered fetchers, then checks each one against what its operator's documentation actually says about robots.txt. At the end you'll find robots.txt examples you can copy for each goal.

The three types of AI crawlers, defined

  • Training crawlers collect web content as candidate training data for generative models. Examples: GPTBot, ClaudeBot.
  • Search indexing crawlers collect web pages and index them as citation candidates for AI search answers. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
  • User-triggered agents are fetchers that retrieve a specific page at the moment a user asks a question or assigns a task. Examples: ChatGPT-User, Claude-User, Perplexity-User.
  • Control tokens are names that send no requests of their own and only set, inside robots.txt, how already-collected content may be used. Example: Google-Extended.
Three kinds of AI crawlers, three different robots.txt scopes Training GPTBot ClaudeBot Content may train models Follows robots.txt Blocking keeps search citations Search indexing OAI-SearchBot Claude-SearchBot PerplexityBot Citation pool for AI answers Follows robots.txt Blocking drops search answers User-triggered ChatGPT-User Claude-User Perplexity-User Fetches page when user asks OpenAI, Perplexity: may not apply Claude-User stops if blocked Control token: Google-Extended Sends no requests. Controls only Gemini training and grounding use of Google-crawled content; no effect on ranking
The three AI crawler types plus the control token. Sources: OpenAI, Anthropic, Perplexity, and Google official documentation (2026).

Nine AI bots compared: purpose and robots.txt behavior

The "robots.txt" column repeats the operator's own documented wording. "Follows" means the operator states that the bot complies. "May not apply" means the operator says robots.txt rules may not apply.

NameOperatorTypeWhat it doesrobots.txtResult of blocking
GPTBotOpenAITrainingCollects content that may be used to train generative foundation modelsFollowsSignals that future content should not be used for training
OAI-SearchBotOpenAISearch indexingSurfaces sites in ChatGPT search featuresFollowsNot shown in ChatGPT search answers; may still appear as navigational links
ChatGPT-UserOpenAIUser-triggeredPerforms user actions in ChatGPT and Custom GPTsMay not applyBlocking access is not guaranteed; not used to decide search visibility
ClaudeBotAnthropicTrainingCollects web content that could contribute to model trainingFollowsSignals that future content should be excluded from training data
Claude-SearchBotAnthropicSearch indexingNavigates the web to improve search result quality for usersFollowsCan't index for search; visibility and accuracy in search results may drop
Claude-UserAnthropicUser-triggeredRetrieves websites when users ask Claude questionsFollowsNo retrieval for user questions; user-directed search visibility may drop
PerplexityBotPerplexitySearch indexingSurfaces and links sites in Perplexity search resultsFollowsPath into search results closed; not used to collect training data
Perplexity-UserPerplexityUser-triggeredVisits pages to answer user questionsGenerally ignoresPages may still be fetched
Google-ExtendedGoogleControl tokenSets whether content is used for Gemini training and groundingThe token itself is a robots.txt ruleNo effect on Google Search inclusion or ranking

What the official documentation says

OpenAI's documentation spells out each bot's role. GPTBot collects content that may be used to train generative foundation models, OAI-SearchBot is for search, and because the two settings are independent, a site can allow OAI-SearchBot while blocking GPTBot.[1] For ChatGPT-User, OpenAI says the actions are user-initiated, so robots.txt rules may not apply, and that this agent is not used to decide whether a site appears in search.[1] The same page notes that after you edit robots.txt, search can take about 24 hours to reflect the change (OpenAI, 2026).[1]

Anthropic uses the same three-way structure but handles the user-triggered agent differently. Its help article says Anthropic's bots honor industry-standard robots.txt directives, and that blocking Claude-User prevents Claude from retrieving site content in response to user queries, which may reduce the site's visibility in user-directed search.[2] The article also says Anthropic supports the non-standard Crawl-delay extension and advises against IP blocking, because blocked bots can't read robots.txt and the opt-out can't be guaranteed (Anthropic, 2026).[2]

Perplexity states that PerplexityBot exists to surface sites in search results and is not used to crawl content for AI foundation models. Perplexity-User is a fetcher that visits pages to answer user questions, and Perplexity says it generally ignores robots.txt rules. Changes can take up to 24 hours to take effect (Perplexity, 2026).[3]

Google defines Google-Extended as a standalone product token. It has no separate HTTP user-agent string; Google's existing crawlers do the collecting, and the token controls whether that content is used to train Gemini models and for grounding in the Gemini app and Vertex AI. The documentation says it has no effect on inclusion in Google Search and is not a ranking signal (Google, 2026).[4] Google says its common crawlers always obey robots.txt during automatic crawls[6], while a separate page says user-triggered fetchers, including Google-Agent, generally ignore robots.txt because a user requested the fetch (Google, 2026).[5]

Why the distinction matters: causes and effects

Most blocking accidents start with a single User-agent: * line. A site owner who wants to stop training adds a blanket block, and the search indexing crawlers get shut out along with the training ones. The damage differs by engine. OpenAI says sites that block OAI-SearchBot won't appear in ChatGPT search answers, so a training opt-out turns into giving up ChatGPT search citations.[1]

The reverse mistake is just as common: a site blocks all three companies' bots by name and assumes AI can no longer read its pages. OpenAI and Perplexity have already said robots.txt may not apply to their user-triggered fetchers[1][3], and Google's user-triggered fetchers take the same position.[5] RFC 9309, the standard itself, states that robots.txt rules are not a form of access authorization (IETF, 2022).[7] Sites that need to keep user-triggered fetchers out have to use tools outside robots.txt, such as login walls, server response codes, and WAF rules. For how CDN and WAF defaults treat these bots, see the CDN and WAF default settings AI crawler audit checklist.

For GEO, the takeaway is simple. If you want AI search citations, keep the three search indexing bots open, and apply your training opt-out only through the training bot names and Google-Extended. The two goals don't conflict. What is GEO explains why AI search citations matter, and the Global GEO and AEO landscape 2026 covers the wider market.

How to configure robots.txt by goal

GoalTraining (GPTBot, ClaudeBot)Search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot)User-triggered (ChatGPT-User, Claude-User, Perplexity-User)Google-Extended
Maximize AI visibilityAllowAllowAllowAllow
Opt out of training, keep search citationsBlockAllowAllowBlock
Opt out of AI entirelyBlockBlockrobots.txt alone is not enough; add server-side measuresBlock

"Opt out of training, keep search citations" is the right default for most brand sites. Here is that configuration:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow search indexing crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Allow user-triggered fetchers
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Roll it out in this order:

  1. Check the current file: Open /robots.txt at your site root and look for Disallow: / under a User-agent: * group. Anthropic notes that each subdomain needs its own robots.txt.[2]
  2. Write a group per bot: Give each bot name its own group, as in the example. When a group names a bot, that bot follows that group (IETF, 2022).[7]
  3. Wait for it to apply: Allow up to about 24 hours, per OpenAI and Perplexity.[1][3]
  4. Check server logs: Compare incoming IPs against the operators' published IP lists to filter out impersonators. OpenAI publishes per-bot IP lists as JSON, and Anthropic publishes its bot IP list.[1][2]
  5. Check rendering: Even when allowed, bots can't read body text that only renders through JavaScript. Use JavaScript rendering and the AI crawler blind spot to confirm the content is in the raw HTML.

The newer shared training opt-out token, __usage_ai-training, is a purpose token meant to replace company-specific names. Its syntax and when each company adopts it are covered in the shared robots.txt AI training opt-out token, and the visibility tradeoff between allowing and blocking is in the AI crawler management guide. If your infrastructure blocks bots by default, as Cloudflare can, read Cloudflare's default AI crawler block and its GEO impact. For opting out of Google's AI features, see Google AI Mode publisher opt-out strategy.

Summary

AI crawlers come in three types: training, search indexing, and user-triggered. Google-Extended is a control token, not a crawler. Operators state that their training and search indexing crawlers follow robots.txt, but user-triggered fetchers vary by company: robots.txt may not apply to ChatGPT-User and Perplexity-User, while Claude-User is documented to stop fetching when blocked. To keep search citations and opt out of training only, block GPTBot, ClaudeBot, and Google-Extended by name and leave the rest open. Then wait 24 hours and confirm the actual behavior in your server logs.

Frequently asked questions

Q.What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's generative models. OAI-SearchBot crawls so a site can appear in ChatGPT's search features. The two settings are independent, so you can block GPTBot and still allow OAI-SearchBot (OpenAI, 2026).
Q.If I block ChatGPT-User in robots.txt, can ChatGPT still open my pages?
There is no guarantee it can't. OpenAI says ChatGPT-User acts on behalf of users, so robots.txt rules may not apply. OpenAI also says ChatGPT-User is not used to decide whether a site appears in search (OpenAI, 2026).
Q.Does Claude-User ignore robots.txt too?
Anthropic says otherwise. Its bots follow robots.txt directives, and blocking Claude-User means Claude can't retrieve your content when a user asks about it, which may reduce your visibility in user-directed search (Anthropic, 2026).
Q.Will blocking Google-Extended remove my site from Google Search?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. The token controls whether content is used to train Gemini models and for grounding in the Gemini app and Vertex AI (Google, 2026).
Q.Does blocking PerplexityBot remove my site from Perplexity answers entirely?
Blocking PerplexityBot closes the path into search results. Perplexity-User, however, visits pages when a user asks a question, and Perplexity says it generally ignores robots.txt, so your pages may still be fetched (Perplexity, 2026).
Q.How long do robots.txt changes take to apply?
OpenAI says search updates take about 24 hours, and Perplexity says up to 24 hours. RFC 9309 also recommends that crawlers not use a cached robots.txt for more than 24 hours (OpenAI, 2026; Perplexity, 2026; IETF, 2022).

Sources

  1. [1] ↑Overview of OpenAI Crawlers — OpenAI
  2. [2] ↑Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
  3. [3] ↑Perplexity Crawlers — Perplexity
  4. [4] ↑Google's common crawlers — Google Search Central
  5. [5] ↑Google's user-triggered fetchers — Google Search Central
  6. [6] ↑Overview of Google crawlers and fetchers — Google Search Central
  7. [7] ↑RFC 9309: Robots Exclusion Protocol — IETF

This document was last edited on Nov 20, 2026. WikiAP content is compiled from public primary sources and updated for accuracy.