Managing AI Crawlers — Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offs
How to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt — based on OpenAI's and Google's official documentation.
For a brand to be cited in generative search, an AI engine's crawler first has to be able to read the page. Yet many sites block AI bots wholesale in robots.txt because "we don't want AI training on our content," and in doing so they also throw away every chance of being cited in search answers. The heart of AI crawler management is to distinguish training bots from search/citation bots and allow or block them selectively.
The problem: when "blocking AI" becomes "blocking search citation"
AI crawlers are not a single thing. OpenAI alone splits roles across GPTBot, which trains the model; OAI-SearchBot, which builds ChatGPT's search answers; and ChatGPT-User, which acts when a user makes a direct request (OpenAI documentation).[1] Block every bot with User-agent: * without knowing this, and you sever search citation along with training.
The cause is simple: operators write robots.txt without distinguishing what each bot name does. The impact is direct — when ChatGPT or Perplexity generates an answer, that site is excluded from the citation pool entirely, and only competitors show up. So the action is clear: identify each bot's purpose, then allow only the bots you need for visibility.
Identifying the major AI crawlers — training vs. search/citation
The dividing line for bots is "what do you lose if this bot is blocked?" Blocking a training bot has almost no effect on search exposure, but blocking a search/citation bot makes you vanish from that engine's answers.
| Crawler | Operator | Purpose | Impact if blocked |
|---|---|---|---|
| GPTBot | OpenAI | Foundation model training | Excluded from training data (no effect on search exposure) |
| OAI-SearchBot | OpenAI | Generating ChatGPT search answers | Excluded from ChatGPT search citations |
| ChatGPT-User | OpenAI | User-triggered requests | Does not honor robots.txt; may still operate when blocked |
| ClaudeBot | Anthropic | Model training | Excluded from training data |
| Claude-SearchBot | Anthropic | Search indexing | Excluded from Claude search citations |
| PerplexityBot | Perplexity | Search indexing | Excluded from Perplexity citations |
| Google-Extended | Controls Gemini/Vertex AI training (token) | Excluded from training (no effect on search rankings) |
Anthropic documents the training bot ClaudeBot, the user-request bot Claude-User, and the search-indexing bot Claude-SearchBot as separate user-agents (ALM Corp).[3] That means you can express the request "no to AI training, but yes to search citation" precisely in robots.txt.
The trade-off: block it, and the citation "source" disappears
The most important trade-off is blocking search/citation bots. Block OAI-SearchBot and your site does not appear in ChatGPT's search answers, and an opted-out site is not featured in answers (OpenAI documentation).[1] The same holds for PerplexityBot and Claude-SearchBot. Citation happens only on pages a bot has read, so block the bot and the very source of the citation is gone.
Conversely, blocking training bots is low-cost. Google-Extended is in fact not a crawler but a robots.txt token, and it governs only the use of your content for Gemini/Vertex AI training. Google has officially confirmed that Google-Extended is not a ranking signal, and blocking it preserves your search rankings and your eligibility to appear in AI Overviews (Google crawler overview).[2] Blocking GPTBot and ClaudeBot likewise filters out training with no visibility loss, as long as you separately allow the search/citation bots.
That said, a complete block is not guaranteed. Requests that a user directly triggers — like ChatGPT-User, Claude-User, and Perplexity-User — may not follow robots.txt (Contently).[4] robots.txt is a "polite request," not a technical firewall, so if you need strict blocking you must pair it with additional measures at the server or WAF layer.
A practical robots.txt example — "refuse training, allow citation"
A common pattern that maximizes visibility while refusing training use looks like this.
# Block model-training bots
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow search/citation bots
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Default (keep search crawlers like Googlebot allowed)
User-agent: *
Allow: /
Conversely, if you want to maximize visibility by allowing training too, simply leave the blocking block empty. Whatever the strategy, the starting point is the same — identify the bot names and their purposes accurately, then decide. This robots.txt design is a fundamental of technical GEO and the starting point of the crawlability check covered in the Tech GEO and Content GEO framework; once access is granted, it must be handled together with AI-citable content structure to actually lead to citations. For checking ChatGPT exposure, see ChatGPT brand visibility.
Summary
The essence of AI crawler management is not "to block AI or not" but whether you control it separately by purpose. Training bots like GPTBot, ClaudeBot, and Google-Extended cost little search visibility when blocked, but block search/citation bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot and the citation source disappears from that engine's answers. And because user-triggered bots may not follow robots.txt, robots.txt should be seen not as a catch-all blocking tool but as the first button to fasten in an exposure strategy. Identify the bot names, separate training from citation, and allow only the bots you need — these three steps are the path to holding visibility and data control at once.
Frequently asked questions
- Not necessarily. GPTBot is for model training, while citations in ChatGPT's search answers are handled by OAI-SearchBot. If you want search exposure, you should allow OAI-SearchBot even if you block GPTBot.
- Training and search bots respect robots.txt, but bots that a user directly triggers — like ChatGPT-User or Claude-User — may not follow robots.txt. If you want a complete block, you need additional measures at the server or WAF level.
- No. Google has officially confirmed that Google-Extended is not a ranking signal. Blocking it preserves both your search rankings and your eligibility to appear in AI Overviews — it only restricts use for Gemini training.
- Only partially. PerplexityBot follows robots.txt, but a path like Perplexity-User that performs live search may not obey the blocking rules, so your content can still appear.
- If visibility is the goal, allow the search/citation bots first (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot). If you object to training use, you can selectively block only GPTBot, ClaudeBot, and Google-Extended.
Q.If I block GPTBot, does my brand disappear from ChatGPT?
Q.If I block it in robots.txt, can AI never read my site at all?
Q.If I block Google-Extended, do my Google Search rankings drop?
Q.If I block PerplexityBot, am I dropped from Perplexity's answers?
Q.Which bots should I allow first?
Sources
Related documents
- Tech GEO + Content GEO — A Two-Axis Method Linking Diagnosis and CreationA framework that splits GEO into a technical diagnostic axis (Technical GEO) and a content creation axis (Content GEO). It lays out what each axis checks and executes, and how the two connect, with a side-by-side comparison table.
- How to Get Your Brand Surfaced in ChatGPTChatGPT builds answers from pretraining data and live web search. This piece lays out how to surface your brand in those answers by allowing GPTBot, structuring content so it can be cited, and consolidating your entity.
- Content Structure That Gets Cited in AI Answers — Writing for ExtractabilityThe writing AI cites is not the same as writing that reads well. How to raise extractability through citable units, answer-first placement, question-answer structure, and tables, lists, and definitions — grounded in GEO research and a practical checklist.