Cloudflare's Default AI Crawler Block, September 15, 2026: Will Your Site Vanish from AI Answers?

"Now that Cloudflare's default AI crawler block is in force as of September 15, 2026, does our site disappear from AI answers?" The policy went public on July 1, 2026, and the defaults took effect on September 15 (Cloudflare, 2026).^[1] For most sites the answer is no, with conditions attached. The defaults do not land on all three categories the same way, they split on whether a page carries ads, and the outcome changes depending on whether a crawler has separated its purposes. This page settles the question in one place: the exact scope of what took effect, the paths where citations really do break, and the items to check in your dashboard today.

The 30-second definitions

• Search is crawler behavior that collects or indexes content to answer questions later (Cloudflare, 2026).^[1]
• Agent is automated behavior that handles something on a person's behalf, usually in real time (Cloudflare, 2026).^[1]
• Training is crawling that takes content to train or fine-tune a model, and the defining trait is that the data goes into training the model (Cloudflare, 2026).^[1]
• Mixed-purpose crawlers are crawlers that serve two or more purposes under a single user agent, and they follow the most restrictive rule that applies (Cloudflare, 2026).^[1]
• Blocked by default describes the state where the edge refuses the request because the site owner configured nothing at all.

Defaults by category, and what each one costs in visibility

The three definitions, the defaults, and the rule that mixed-purpose crawlers inherit the strictest applicable setting are all written into Cloudflare's announcement documents (Cloudflare, 2026).^[1] Press coverage interpreted the change as pressure on AI companies: separate your training crawlers or face default blocks from September 15 (MLQ News, 2026).^[4]

First, sort out whether your domain is in scope

Cloudflare's press release and TechCrunch's reporting agree on both points: the scope covers new domains, new customers, and existing Free plans, and anyone who wanted the old behavior could adjust it in security settings before September 15 (Cloudflare, 2026; TechCrunch, 2026).^[2][3] The setting remains editable in the dashboard after that date (Cloudflare, 2026).^[1]

Where these defaults came from

As of July 2025, crawls per referred visitor came to 38,065.7 for Anthropic, 1,091.4 for OpenAI, 194.8 for Perplexity, 40.7 for Microsoft, and 5.4 for Google (Cloudflare, 2025).^[5] Cloudflare presented that gap between crawling and referred traffic as the background for redesigning the defaults. It also said more than 50% of crawl traffic from good bots goes to re-fetching pages that have not changed (Cloudflare, 2026).^[6] The same announcement counted more than 50% of traffic online as non-human and put more than 20% of the web behind Cloudflare's network (Cloudflare, 2026).^[6] That is why one line of default configuration reaches a wide range of sites at once.

Where citations actually break

The defaults sound large because of the shorthand, "Cloudflare blocks AI crawlers." The real behavior is narrower. Search is allowed by default, so for documentation-style pages that carry no ads, nothing changed on either side of September 15. There are two spots to watch.

First, the Agent block on ad pages. Cloudflare blocks Agent and Training by default on pages that show ads (Cloudflare, 2026).^[1] Agent covers real-time behavior such as the fetch a chatbot performs when it opens a page mid-conversation. Your index entry survives, but if the source check fails inside the current conversation, an up-to-date citation or link check can drop out. Pages that change often, carrying prices, stock, schedules, or specifications, are the most sensitive to this.

Second, rule inheritance among mixed-purpose crawlers. Cloudflare's documentation states that when one user agent serves two or more purposes, the most restrictive rule applies (Cloudflare, 2026).^[1] So even with Search allowed, a Training block that lands on the same user agent can reach search collection as well. What matters here is which purposes a given search engine crawler actually performs. That has to be checked against each crawler's own published documentation, because Cloudflare's category description alone does not establish whether a particular crawler doubles as a training crawler. In practice, look first for a robots.txt token or another control the operator provides before reaching for an edge block. Managing AI crawlers and the allow-or-block trade-offs walks through this judgment call bot by bot.

Line up robots.txt declarations with edge blocking

Cloudflare added a use= field to robots.txt Content Signals to state how content may be used. The three values are immediate, which allows interaction but no storage or reuse, reference, which allows indexing and quoting with a link back, and full, which extends to summarizing and reproducing (Cloudflare, 2026).^[1]

Content-Signal: search=yes,ai-train=no,use=reference

The worst state is a declaration that contradicts the actual block. Write "search allowed" in robots.txt while the edge turns away mixed-purpose crawlers, and a rule-following crawler reads permission while its request gets refused. That drags out the hunt for the cause. The declaration is a document that announces intent, the category setting is a switch that stops requests. Both layers need to point the same way. The complete guide to llms.txt covers what extra declaration files like llms.txt do and where they stop.

Options besides blocking

Open everything or block everything are not the only two settings. AI Crawl Control lets you block crawlers one at a time and return a 402 response with a note in it instead of a bare block. The AI Crawl Control announcement states that on an average day Cloudflare customers are already sending over one billion 402 response codes (Cloudflare, 2025).^[7] The product bundles crawler management, AI traffic analytics, robots.txt compliance tracking, and Pay per crawl (Cloudflare, 2026).^[8] Pay per crawl sets a flat per-request price at the domain level, returns 402 Payment Required to crawlers with no intent to pay, and lets a crawler declare its ceiling in advance through a crawler-max-price header. It has run as a private beta, and pricing follows published terms that may change (Cloudflare, 2025).^[9] More recently Cloudflare has shown experiments that pay out on actual usage rather than crawl counts (Cloudflare, 2026).^[6]

Action, five things to check today

Order matters in step 3. Before blocking Training, find out whether that crawler separated its purposes. If it did, you can refuse training and keep the index. If it did not, the price of the block can extend to search visibility.

Step 5 builds the evidence you need to reverse a setting. Rankings and citations may wobble right after a block, and with no baseline it is hard to tell a crawler setting apart from a model update. That takes a tool that repeats the same question set per engine and records whether you were cited. In Korea, Ascent AI, Next-T, LeadGenLab, and BOIDA state that they track visibility in AI answers. Those descriptions come from the vendors themselves, so check the actual feature scope against each one's documentation and demo. Multi-engine measurement takes on the measurement design itself.

Wrap-up

The September 15, 2026 default is not a switch that erases you from AI, it is closer to a switch that splits by purpose. Search, the category that creates citations, stays allowed, and the default block is confined to Agent and Training on ad pages. Most documentation-style pages carry on unchanged. What can be affected is the live fetch on ad pages, plus search collection whenever a Training block applies to a mixed-purpose user agent. Today's job is to check both spots in the dashboard and to point the robots.txt declaration and the edge settings the same way. For wider context, continue with What is GEO and Google AI Mode publisher opt-out and GEO strategy.