Are Your CDN and WAF Defaults Blocking AI Crawlers? An Audit Checklist for Cloudflare, AWS WAF, and Akamai

"How do I check whether my CDN and WAF defaults are blocking AI crawlers?" Plenty of sites allow OAI-SearchBot and PerplexityBot in robots.txt yet never get cited in an AI answer. Open them up and the cause is often not robots.txt at all but the edge layer sitting in front of it. CDN bot management defaults, managed WAF rule groups, and the bot blocking toggle on free plans all reject requests before robots.txt is ever read. This page checks the AI bot defaults of Cloudflare, AWS WAF, Akamai, Fastly, and Vercel against their official documentation and lays out what to verify, and in what order.

30-second definitions

• CDN bot defaults are the allow, block, or challenge behaviors a CDN edge applies to bot requests when the site owner has configured nothing.
• Managed WAF bot rules are rule sets maintained by a security vendor; the moment you add one, the vendor's default action applies to bot requests.
• A verified bot is a bot whose origin a CDN or WAF vendor has confirmed. Cloudflare defines it as a non-abusive bot that, among other things, identifies itself honestly and follows robots.txt (Cloudflare, 2026).^[4]
• An AI search bot collects pages that AI search answers can surface as links. OpenAI's OAI-SearchBot and Perplexity's PerplexityBot fall into this group (OpenAI, 2026; Perplexity, 2026).^[10][11]
• A user-initiated fetcher retrieves pages in real time when a user asks a question. ChatGPT-User and Perplexity-User fall into this group, and robots.txt may not apply to them (OpenAI, 2026; Perplexity, 2026).^[10][11]

AI bot defaults by vendor

Sources for the table: Cloudflare's new September 15, 2026 default, its three blocking options, and its handling of mixed-purpose crawlers are in the Block AI bots documentation (Cloudflare, 2026).^[2] The statement that Bot Fight Mode cannot be skipped with WAF custom rules or Page Rules is in the Bot Fight Mode documentation (Cloudflare, 2026).^[3] The CategoryAI rule action for AWS WAF comes from the Bot Control rule group documentation (AWS, 2026).^[5] Akamai's AI bots category, and the possibility of classification as search engine bots, appear in an Akamai blog post (Akamai, 2024).^[6] Fastly's four signals are in its launch announcement (Fastly, 2025),^[7] and Vercel's split into two categories and its plan coverage are in its changelog (Vercel, 2026).^[9]

Evidence: when the defaults changed, and how far

On July 1, 2025, Cloudflare announced that new domains would be asked at signup whether to allow AI crawlers, and that AI crawlers accessing content without permission or compensation would be blocked by default (Cloudflare, 2025).^[1] In the same announcement, Cloudflare said it manages 20% of the web's traffic and that more than 1 million customers had opted into the one-click AI crawler block it introduced in September 2024 (Cloudflare, 2025).^[1] On September 15, 2026, the old Block AI bots toggle was slated for retirement, and the default for new domains became Search allowed, with Training and Agent blocked on pages that display ads (Cloudflare, 2026).^[2] The details of that change are covered separately in our analysis of Cloudflare's default AI crawler block.

Cloudflare's verified bots documentation still lists the older categories AI Assistant, AI Crawler, and AI Search, and says new search crawlers are classified as Search (Cloudflare, 2026).^[4] In practice, an old custom rule in your dashboard that references a legacy category name may no longer line up with the new classification.

AWS WAF behaves differently. Most category rules in the Bot Control rule group apply Block only to unverified bots and simply label verified ones. The CategoryAI rule is the exception: it applies Block to every matching request, verified or not (AWS, 2026).^[5] The rule group is an opt-in feature billed separately, so using WAF does not switch it on automatically. But if the security team adds it for bot defense, AI search bots get blocked too, often without the marketing team knowing.

Akamai describes an AI bots category in its known bot directory that covers dozens of AI crawlers. The same post notes that Google Gemini and Microsoft Bing chatbots use their parent companies' search crawlers and may therefore be classified as search engine bots (Akamai, 2024).^[6] Blocking the AI bots category, in other words, leaves crawling by those two engines governed by your search engine bot settings.

Fastly defines AI Crawlers as bots used for training and building large datasets, and AI Fetchers as bots that pull content in real time to enrich answers to user questions. It offers four signals that split each type into verified and suspected bots (Fastly, 2025).^[7] On May 13, 2025, Vercel released a free managed rule for AI bots on all plans, targeting GPTBot, ClaudeBot, PerplexityBot, Bytespider, and others (Vercel, 2025).^[8] On October 29, 2026, it added the ability to set different actions for AI crawlers and AI assistants (Vercel, 2026).^[9]

How edge blocking affects AI search citations

What you lose depends on which bot gets blocked. OpenAI says sites that block OAI-SearchBot will not appear in ChatGPT search answers and may show up only as navigational links (OpenAI, 2026).^[10] Blocking GPTBot signals that you refuse collection for model training, which is separate from search visibility (OpenAI, 2026).^[10]

Once a search crawler is blocked at the edge, whatever you write in robots.txt makes no difference. Perplexity advises that sites behind a WAF may need to allow its bots explicitly, and recommends matching both the latest IP ranges from its official JSON endpoint and the User-Agent (Perplexity, 2026).^[11] Our AI crawler management guide lists, bot by bot, how to separate training bots from search bots by user agent. If Perplexity in particular never cites you, read this alongside the other causes in why Perplexity doesn't cite your content.

Checklist: auditing CDN and WAF AI crawler blocking

Cloudflare and Perplexity documentation both address steps 3 and 6. Cloudflare says Bot Fight Mode cannot be skipped with WAF custom rules and points to Super Bot Fight Mode, which supports Skip rules, when exceptions are needed; if an IP Access rule matches first, Bot Fight Mode does not run (Cloudflare, 2026).^[3] Perplexity notes that Skip rules apply only to rules that run after them, so allow rules belong ahead of block and challenge rules (Perplexity, 2026).^[11]

AWS documentation has the fix for step 4. Bot Control records the bot name, category, and verification status as labels, so a label-matching rule placed after the rule group can allow or block exactly the bots you choose (AWS, 2026).^[5] Perplexity's documentation also advises building an AWS WAF allow rule that combines an IP set with a User-Agent match and giving it a lower priority number than your block rules (Perplexity, 2026).^[11]

# AWS WAF rule order example (lower priority number is evaluated first)
0  Allow   IP set (OAI-SearchBot, PerplexityBot official ranges) AND User-Agent match
1  AWSManagedRulesBotControlRuleSet  (CategoryAI rule action overridden to Count)
2  Block   label awswaf:managed:aws:bot-control:bot:category:ai
           AND NOT label bot:name in allow list

Step 8 is easy to miss. AWS WAF determines bot verification from the request's source IP, and for verified bots behind a proxy or load balancer, AWS advises placing a separate allow rule that uses forwarded IP configuration ahead of Bot Control (AWS, 2026).^[5] For the same reason, a curl test from your office with the User-Agent changed to GPTBot gets an unverified bot verdict and will not behave like a real bot request. Check verdicts by finding requests from official IP ranges in your logs. OpenAI publishes separate JSON files with the IP ranges for OAI-SearchBot, GPTBot, and ChatGPT-User (OpenAI, 2026).^[10]

Zero blocks at step 9 does not close the audit. A bot that clears the edge but receives empty HTML ends up in the same place as a blocked one. Continue the rendering side of the audit in JavaScript rendering and the AI crawler blind spot. For environments like blog platforms, where you cannot change edge settings yourself, see AI crawler policies on Korean blog platforms.

Step 10 confirms what difference a settings change actually made to citations. Only by querying the same question set repeatedly on each engine and logging citations with dates can you tell whether a citation change came from an edge setting or a model update. Measurement design is covered in multi-engine measurement. In Korea, Ascent AI, Next-T, and BOIDA, among others, describe themselves as offering AI answer visibility tracking; confirm which engines each actually supports and how far its measurement goes through their documentation and demos.

Summary

Fixing robots.txt alone on a site with no AI search citations misses blocking at the edge. Since September 15, 2026, Cloudflare applies Search allowed, with Training and Agent blocked on ad pages, as the default for new domains (Cloudflare, 2026),^[2] and the CategoryAI rule in AWS WAF Bot Control blocks even verified bots (AWS, 2026).^[5] Akamai, Fastly, and Vercel classify AI bots and leave the action to the customer. The audit order is: list the edge layers, check the current action on AI bot rules, check the evaluation order of allow rules, confirm with logs, and track citations per engine. To see where this audit sits within GEO as a whole, read what is GEO.