Types of AI Crawlers: Search vs. Training vs. User-Triggered Agents, and How robots.txt Applies
GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, ChatGPT-User and four more AI bots sorted into training, search indexing, and user-triggered fetchers, with robots.txt behavior taken from OpenAI, Anthropic, and Google documentation.
Write robots.txt rules without knowing which kind of AI crawler you're dealing with, and you can get the opposite of what you wanted. A site that only meant to opt out of training vanishes from ChatGPT search answers. A site that thought it had blocked everything keeps getting fetched whenever a user asks about it. Both mistakes have one cause. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and Google-Extended have similar names, but they do different jobs and robots.txt covers them differently. This guide sorts those nine names into training crawlers, search indexing crawlers, and user-triggered fetchers, then checks each one against what its operator's documentation actually says about robots.txt. At the end you'll find robots.txt examples you can copy for each goal.
The three types of AI crawlers, defined
- Training crawlers collect web content as candidate training data for generative models. Examples: GPTBot, ClaudeBot.
- Search indexing crawlers collect web pages and index them as citation candidates for AI search answers. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User-triggered agents are fetchers that retrieve a specific page at the moment a user asks a question or assigns a task. Examples: ChatGPT-User, Claude-User, Perplexity-User.
- Control tokens are names that send no requests of their own and only set, inside robots.txt, how already-collected content may be used. Example: Google-Extended.
Nine AI bots compared: purpose and robots.txt behavior
The "robots.txt" column repeats the operator's own documented wording. "Follows" means the operator states that the bot complies. "May not apply" means the operator says robots.txt rules may not apply.
| Name | Operator | Type | What it does | robots.txt | Result of blocking |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Collects content that may be used to train generative foundation models | Follows | Signals that future content should not be used for training |
| OAI-SearchBot | OpenAI | Search indexing | Surfaces sites in ChatGPT search features | Follows | Not shown in ChatGPT search answers; may still appear as navigational links |
| ChatGPT-User | OpenAI | User-triggered | Performs user actions in ChatGPT and Custom GPTs | May not apply | Blocking access is not guaranteed; not used to decide search visibility |
| ClaudeBot | Anthropic | Training | Collects web content that could contribute to model training | Follows | Signals that future content should be excluded from training data |
| Claude-SearchBot | Anthropic | Search indexing | Navigates the web to improve search result quality for users | Follows | Can't index for search; visibility and accuracy in search results may drop |
| Claude-User | Anthropic | User-triggered | Retrieves websites when users ask Claude questions | Follows | No retrieval for user questions; user-directed search visibility may drop |
| PerplexityBot | Perplexity | Search indexing | Surfaces and links sites in Perplexity search results | Follows | Path into search results closed; not used to collect training data |
| Perplexity-User | Perplexity | User-triggered | Visits pages to answer user questions | Generally ignores | Pages may still be fetched |
| Google-Extended | Control token | Sets whether content is used for Gemini training and grounding | The token itself is a robots.txt rule | No effect on Google Search inclusion or ranking |
What the official documentation says
OpenAI's documentation spells out each bot's role. GPTBot collects content that may be used to train generative foundation models, OAI-SearchBot is for search, and because the two settings are independent, a site can allow OAI-SearchBot while blocking GPTBot.[1] For ChatGPT-User, OpenAI says the actions are user-initiated, so robots.txt rules may not apply, and that this agent is not used to decide whether a site appears in search.[1] The same page notes that after you edit robots.txt, search can take about 24 hours to reflect the change (OpenAI, 2026).[1]
Anthropic uses the same three-way structure but handles the user-triggered agent differently. Its help article says Anthropic's bots honor industry-standard robots.txt directives, and that blocking Claude-User prevents Claude from retrieving site content in response to user queries, which may reduce the site's visibility in user-directed search.[2] The article also says Anthropic supports the non-standard Crawl-delay extension and advises against IP blocking, because blocked bots can't read robots.txt and the opt-out can't be guaranteed (Anthropic, 2026).[2]
Perplexity states that PerplexityBot exists to surface sites in search results and is not used to crawl content for AI foundation models. Perplexity-User is a fetcher that visits pages to answer user questions, and Perplexity says it generally ignores robots.txt rules. Changes can take up to 24 hours to take effect (Perplexity, 2026).[3]
Google defines Google-Extended as a standalone product token. It has no separate HTTP user-agent string; Google's existing crawlers do the collecting, and the token controls whether that content is used to train Gemini models and for grounding in the Gemini app and Vertex AI. The documentation says it has no effect on inclusion in Google Search and is not a ranking signal (Google, 2026).[4] Google says its common crawlers always obey robots.txt during automatic crawls[6], while a separate page says user-triggered fetchers, including Google-Agent, generally ignore robots.txt because a user requested the fetch (Google, 2026).[5]
Why the distinction matters: causes and effects
Most blocking accidents start with a single User-agent: * line. A site owner who wants to stop training adds a blanket block, and the search indexing crawlers get shut out along with the training ones. The damage differs by engine. OpenAI says sites that block OAI-SearchBot won't appear in ChatGPT search answers, so a training opt-out turns into giving up ChatGPT search citations.[1]
The reverse mistake is just as common: a site blocks all three companies' bots by name and assumes AI can no longer read its pages. OpenAI and Perplexity have already said robots.txt may not apply to their user-triggered fetchers[1][3], and Google's user-triggered fetchers take the same position.[5] RFC 9309, the standard itself, states that robots.txt rules are not a form of access authorization (IETF, 2022).[7] Sites that need to keep user-triggered fetchers out have to use tools outside robots.txt, such as login walls, server response codes, and WAF rules. For how CDN and WAF defaults treat these bots, see the CDN and WAF default settings AI crawler audit checklist.
For GEO, the takeaway is simple. If you want AI search citations, keep the three search indexing bots open, and apply your training opt-out only through the training bot names and Google-Extended. The two goals don't conflict. What is GEO explains why AI search citations matter, and the Global GEO and AEO landscape 2026 covers the wider market.
How to configure robots.txt by goal
| Goal | Training (GPTBot, ClaudeBot) | Search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot) | User-triggered (ChatGPT-User, Claude-User, Perplexity-User) | Google-Extended |
|---|---|---|---|---|
| Maximize AI visibility | Allow | Allow | Allow | Allow |
| Opt out of training, keep search citations | Block | Allow | Allow | Block |
| Opt out of AI entirely | Block | Block | robots.txt alone is not enough; add server-side measures | Block |
"Opt out of training, keep search citations" is the right default for most brand sites. Here is that configuration:
# Block training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow search indexing crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Allow user-triggered fetchers
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
Roll it out in this order:
- Check the current file: Open
/robots.txtat your site root and look forDisallow: /under aUser-agent: *group. Anthropic notes that each subdomain needs its own robots.txt.[2] - Write a group per bot: Give each bot name its own group, as in the example. When a group names a bot, that bot follows that group (IETF, 2022).[7]
- Wait for it to apply: Allow up to about 24 hours, per OpenAI and Perplexity.[1][3]
- Check server logs: Compare incoming IPs against the operators' published IP lists to filter out impersonators. OpenAI publishes per-bot IP lists as JSON, and Anthropic publishes its bot IP list.[1][2]
- Check rendering: Even when allowed, bots can't read body text that only renders through JavaScript. Use JavaScript rendering and the AI crawler blind spot to confirm the content is in the raw HTML.
The newer shared training opt-out token, __usage_ai-training, is a purpose token meant to replace company-specific names. Its syntax and when each company adopts it are covered in the shared robots.txt AI training opt-out token, and the visibility tradeoff between allowing and blocking is in the AI crawler management guide. If your infrastructure blocks bots by default, as Cloudflare can, read Cloudflare's default AI crawler block and its GEO impact. For opting out of Google's AI features, see Google AI Mode publisher opt-out strategy.
Summary
AI crawlers come in three types: training, search indexing, and user-triggered. Google-Extended is a control token, not a crawler. Operators state that their training and search indexing crawlers follow robots.txt, but user-triggered fetchers vary by company: robots.txt may not apply to ChatGPT-User and Perplexity-User, while Claude-User is documented to stop fetching when blocked. To keep search citations and opt out of training only, block GPTBot, ClaudeBot, and Google-Extended by name and leave the rest open. Then wait 24 hours and confirm the actual behavior in your server logs.
Frequently asked questions
- GPTBot collects content that may be used to train OpenAI's generative models. OAI-SearchBot crawls so a site can appear in ChatGPT's search features. The two settings are independent, so you can block GPTBot and still allow OAI-SearchBot (OpenAI, 2026).
- There is no guarantee it can't. OpenAI says ChatGPT-User acts on behalf of users, so robots.txt rules may not apply. OpenAI also says ChatGPT-User is not used to decide whether a site appears in search (OpenAI, 2026).
- Anthropic says otherwise. Its bots follow robots.txt directives, and blocking Claude-User means Claude can't retrieve your content when a user asks about it, which may reduce your visibility in user-directed search (Anthropic, 2026).
- No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. The token controls whether content is used to train Gemini models and for grounding in the Gemini app and Vertex AI (Google, 2026).
- Blocking PerplexityBot closes the path into search results. Perplexity-User, however, visits pages when a user asks a question, and Perplexity says it generally ignores robots.txt, so your pages may still be fetched (Perplexity, 2026).
- OpenAI says search updates take about 24 hours, and Perplexity says up to 24 hours. RFC 9309 also recommends that crawlers not use a cached robots.txt for more than 24 hours (OpenAI, 2026; Perplexity, 2026; IETF, 2022).
Q.What is the difference between GPTBot and OAI-SearchBot?
Q.If I block ChatGPT-User in robots.txt, can ChatGPT still open my pages?
Q.Does Claude-User ignore robots.txt too?
Q.Will blocking Google-Extended remove my site from Google Search?
Q.Does blocking PerplexityBot remove my site from Perplexity answers entirely?
Q.How long do robots.txt changes take to apply?
Sources
- [1] ↑Overview of OpenAI Crawlers — OpenAI
- [2] ↑Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
- [3] ↑Perplexity Crawlers — Perplexity
- [4] ↑Google's common crawlers — Google Search Central
- [5] ↑Google's user-triggered fetchers — Google Search Central
- [6] ↑Overview of Google crawlers and fetchers — Google Search Central
- [7] ↑RFC 9309: Robots Exclusion Protocol — IETF
Related documents
- Managing AI Crawlers: Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offsHow to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt: based on OpenAI's and Google's official documentation.
- __usage_ai-training Explained: The Shared robots.txt AI Training Opt-Out Token and Its GEO ImpactOpenAI, Google, Microsoft, and Anthropic have agreed to honor a shared robots.txt token, __usage_ai-training. Here is the syntax, when each company starts applying it, how blocking training affects AI search citations, and the order to change your settings.
- Cloudflare's Default AI Crawler Block, September 15, 2026: Will Your Site Vanish from AI Answers?A breakdown of the Search, Agent, and Training defaults Cloudflare put in force on September 15, 2026. Search crawlers stay allowed, while Agent blocks and mixed-purpose crawler inheritance are the real paths that cut citations in AI answers.
- Are Your CDN and WAF Defaults Blocking AI Crawlers? An Audit Checklist for Cloudflare, AWS WAF, and AkamaiA checklist, built from official vendor documentation, for confirming whether the bot management defaults in Cloudflare, AWS WAF, Akamai, Fastly, and Vercel block GPTBot, OAI-SearchBot, and PerplexityBot, and what that means for AI search citations.
- Do AI Crawlers Run JavaScript? Auditing and Fixing Missing Citations for CSR/SPA ContentGPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. Here is why CSR/SPA content disappears from AI search citations, and the fix sequence from a curl audit to server-side rendering, based on official documentation.
- Google AI Mode and AI Overviews Publisher Opt-Out: The GEO Strategy Dilemma and Decision FrameworkGoogle rolled out AI Mode and AI Overviews opt-out toggles in Search Console in June 2026. Opting out eliminates GEO visibility entirely; staying in means AI consumes content for free while traffic erodes. Here is how the dilemma is structured and how different publisher types should choose.
- What Is GEO: The Definition of Generative Engine Optimization and How It Differs From SEOGEO (Generative Engine Optimization) is the strategy of getting your content cited in answers produced by generative engines like ChatGPT and Perplexity. Here is the definition, how it differs from SEO, and how it works.
- Global GEO/AEO Player Landscape 2026, Monitoring Tools, Agencies, and PlatformsA 2026 landscape that sorts GEO/AEO players into monitoring tools, specialist solutions and agencies, enterprise platforms, and regional players. We compare the leading vendor in each category, founding, headquarters, tracked engines, pricing, and differentiation, against primary sources.