How Paid AI Crawling Affects GEO: AI Citation Strategy After Pay per Crawl and RSL
A single table comparing Cloudflare Pay per crawl and Pay per use, RSL 1.0, Microsoft PCM, and Perplexity Comet Plus, plus what content monetization does to AI answer citations and GEO, and the page-level rules a brand should set.
The effect of paid AI crawling on GEO and AI citations fits in one sentence: a page you charge for drops out of the answers of every engine that doesn't pay. Cloudflare announced Pay per crawl, a per-request fee for crawlers, on July 1, 2025 (Cloudflare, 2025).[1] A year later, in July 2026, it unveiled Pay per use experiments that compensate publishers for how much their content shows up in answers rather than how often it gets crawled (Cloudflare, 2026).[2] In between, in December 2025, RSL 1.0 became a standard for declaring license terms that machines can read (Editor and Publisher, 2025).[6] This guide sorts five monetization models into one table, shows how each one reshapes the pool of citation candidates, and lays out what brands and publishers should open up and what they should charge for, page by page.
30-second definitions
- Pay per crawl is Cloudflare's billing method that sets a domain-level price on each AI crawler request and returns a 402 response when the crawler doesn't pay (Cloudflare, 2025).[1]
- Pay per use is Cloudflare's experimental usage-based model that compensates publishers when their content is actually used in an AI answer or search result (Cloudflare, 2026).[2]
- RSL (Really Simple Licensing) is an open standard for declaring the terms under which AI systems and automated agents may access, use, and license content (RSL, 2025).[5]
- A content marketplace is a platform that connects many publishers with many AI buyers and handles licensing and settlement without one-off contracts (Microsoft Advertising, 2026).[7]
- A citation candidate pool is the set of pages a given AI engine can fetch and use as evidence when it builds an answer.
AI content monetization models compared
| Model | Operator | Basis for payment | Declaration or enforcement | Status | Effect on AI citations |
|---|---|---|---|---|---|
| Pay per crawl | Cloudflare | Each crawl request | 402 response at the edge, pricing headers | Private beta | Non-paying engines stop collecting the source text |
| Pay per use | Cloudflare and AI companies | Number of times used in answers or search results | Experiments between participating publishers and AI companies | Experimental | Participating engines report citation data |
| RSL 1.0 | RSL Technical Steering Committee, RSL Collective | Declared license terms | robots.txt License directive, XML license file | Finalized standard | Granular declarations can refuse AI search use only |
| Publisher Content Marketplace | Microsoft Advertising | Value delivered, usage-based reporting | Marketplace agreements | Pilot | Copilot responses grounded in contracted content |
| Comet Plus revenue share | Perplexity | Visits, search citations, agent tasks | Joining the publisher program | Announced August 2025 | The citation itself becomes the unit of payment |
Pay per crawl lets a site choose among allow, charge, and block, with Cloudflare acting as Merchant of Record for settlement (Cloudflare, 2025).[1] In the Pay per use experiments, Ceramic.ai pays per query when a publisher's content appears in its search results, and You.com pays when an agent needs a specific piece of premium content (Cloudflare, 2026).[2] Microsoft announced PCM on February 3, 2026. It began as a pilot that grounds Copilot responses in licensed content and has started bringing on Yahoo as a demand-side partner (Microsoft Advertising, 2026).[7] Perplexity says it has opened a $42.5 million distribution pool and shares 80% of subscription revenue with publishers in its program, splitting payouts across three categories: human visits, citations in Comet search answers, and use in AI assistant tasks (Digiday, 2025).[8]
The evidence: crawling is up, referrals are down
| Item | Value (%) | Source |
|---|---|---|
| Training 2024-07 | 72% | (Cloudflare, 2025) |
| Training 2025-07 | 79% | (Cloudflare, 2025) |
| Search 2024-07 | 26% | (Cloudflare, 2025) |
| Search 2025-07 | 17% | (Cloudflare, 2025) |
| User action 2024-07 | 2% | (Cloudflare, 2025) |
| User action 2025-07 | 3.2% | (Cloudflare, 2025) |
The monetization debate starts with the numbers. According to Cloudflare Radar, training rose from 72% of AI crawling in July 2024 to 79% in July 2025, while search fell from 26% to 17% (Cloudflare, 2025).[4] Across the preceding 12 months as a whole, the split was 80% training, 18% search, and 2% user action (Cloudflare, 2025).[4] Crawls per referred visit in July 2025 stood at 38,065.7 for Anthropic, 1,091.4 for OpenAI, and 194.8 for Perplexity (Cloudflare, 2025).[4] Publishers see their source text collected in bulk while the clicks coming back are tiny next to the crawl volume. Charging is an attempt to cover that gap with money.
Cloudflare itself treats crawl counts as a crude way to measure value, which is why it moved toward usage-based compensation. Its stated reason is internal data showing that more than 50% of good-bot crawl traffic goes to re-fetching pages that haven't changed (Cloudflare, 2026).[2]
Three ways monetization changes AI citations
Citation candidate pools split by engine. Under Pay per crawl, a crawler that requests a paid page without signaling intent to pay gets a 402 response and a pricing header. A crawler can send a maximum-price header with its first request, and if the configured price is at or below that cap, it receives the source text with a 200 response (Cloudflare, 2025).[1] The same page can then serve as evidence in the answers of engines that paid and never appear in the answers of engines that didn't. GEO has been built on the assumption that every engine sees the same web. Monetization breaks that assumption, so citation differences have to be measured engine by engine.
The unit of payment becomes the citation. Pay per use, Comet Plus, and PCM all settle on the fact of use in an answer, not on the crawl (Cloudflare, 2026; Digiday, 2025; Microsoft Advertising, 2026).[2][8][7] Under this setup, a publisher's revenue scales with how often AI answers cite it. This is where GEO stops being a PR metric and becomes a revenue metric. Pages built to be cited, with extractable definitions, real tables, and source attributions, become the pages that earn more. How engines pick citations in the first place is covered in How AI chooses its citation sources.
Declarations get more granular. RSL 1.0 defines usage categories such as ai-all, ai-input, and ai-index, and supports combinations that allow listing in search engines while refusing use in AI search (Editor and Publisher, 2025).[6] A site points to an XML license file with a License directive in robots.txt, and an Open License Protocol server issues license tokens (RSL, 2025).[5] Cloudflare has added a use field to Content Signals in robots.txt as well, separating immediate (interaction only), reference (indexing, excerpts, and links), and full (summaries and reproduction) (Cloudflare, 2026).[3] The more granular the declarations, the easier it becomes to end up by accident in a state where a page shows in search but never in AI answers.
Who gains what, and who loses what
| Site type | What monetization gains | What monetization costs | Recommended default |
|---|---|---|---|
| News and specialist media | Payment for use of the source text | Citations and referrals from non-paying engines | Charge for archives only, allow reference on some recent articles |
| Paid reports and databases | Revenue from access itself | The awareness path through cited summaries | Charge for full text, allow summary pages |
| Brands and commerce | Almost nothing | Visibility in product recommendation answers | Free access, use=reference |
| B2B SaaS and services | Almost nothing | Eligibility as a candidate in comparison and recommendation answers | Free access, remove ads from documentation-style pages |
| Communities and UGC | Bargaining power in large-scale contract negotiations | Citations from engines without a contract | Measure contracted and non-contracted engines separately |
There is one test. Does the page's value come from people reading the source, or from having your name show up in AI answers? If it's the first, consider charging. If it's the second, keep it open. Most brand sites fall into the second group. Put product information behind a fee and you collect almost nothing while your name drops out of recommendation answers.
One point is easy to confuse. Starting September 15, 2026, Cloudflare blocks Training and Agent by default and allows Search by default on ad-displaying pages, for new domains among others (Cloudflare, 2026).[3] That is a default block, not a fee. Its scope and the order of checks are covered separately in Cloudflare's default AI crawler block and its GEO impact.
Action plan: a 5-step GEO check for the paid-crawling era
| Step | Task | Pass criterion |
|---|---|---|
| 1 | Classify pages as citation-focused or access-focused | Every major URL pattern has exactly one assigned purpose |
| 2 | Compare CDN charge and block settings against robots.txt declarations | Content Signals, RSL License, and edge settings express the same intent |
| 3 | Explicitly allow citation-focused pages at the use=reference level | Both search indexing and AI excerpts are allowed |
| 4 | Run a separate summary or public abstract page for each paid page | At least 1 citable public URL per paid page |
| 5 | Measure citation changes separately for each engine | Same question set compared per engine, before and after the change |
Step 2 is where things break most often. If robots.txt says allow while the edge applies a fee or a block, compliant crawlers read the page as allowed and their actual requests get a 402 or 403. For the trade-offs of allowing or blocking individual bots, see the AI crawler management guide. For crawler policies on Korean blog platforms, see Korean blog platform AI crawler policies and citations.
Step 5 becomes mandatory once monetization arrives. Each engine has its own citation candidate pool, so what you see in ChatGPT tells you nothing reliable about Perplexity or Copilot. Publishers in the Ceramic search program within the Pay per use experiment receive reports on top queries, the pages and snippets used, and average search rank (Cloudflare, 2026).[2] PCM also provides usage-based reporting (Microsoft Advertising, 2026).[7] Sites outside these programs have to run the same question set on each engine repeatedly and log whether they get cited. Globally, Profound and Peec AI, and in Korea, Ascent AI, Next-T, and BOIDA, among others, position themselves as tracking AI answer visibility by engine. Check each vendor's documentation and demo for the actual feature scope. Measurement design continues in Multi-engine measurement.
Summary
Paid AI crawling is moving from per-crawl fees to usage-based compensation. Pay per crawl is in private beta, and Pay per use and PCM are at the experiment and pilot stage (Cloudflare, 2025; Cloudflare, 2026; Microsoft Advertising, 2026).[1][2][7] For GEO, two things change: each engine sees a different web, and citation counts become the unit of settlement. Brands need to keep citation-focused pages open and align their declarations with their edge settings. Publishers need to leave a citable public summary for every paid page. For background, read What is GEO; for the market as a whole, The 2026 global GEO and AEO landscape; and for a publisher opt-out case, Google AI Mode publisher opt-out strategy.
Related companies
- 넥스트티 (Next-T, OPTIGEO)SEO, GEO, AEO 컨설팅, 자동화
- 보이다 (BOIDA)생성형 검색 최적화(GEO) 솔루션, AI 가시성 측정
- 어센트 AI (ASCENT AI, ListeningMind)인텐트 인텔리전스, GEO
- Peec AIAI 가시성 모니터링 플랫폼
- ProfoundAI 가시성 모니터링 플랫폼
Frequently asked questions
- Crawlers that don't pay get a 402 Payment Required response, so any engine relying on those crawlers can't fetch the page's source text. Without the source text, that engine has nothing to cite the page from. The safe move is to limit fees to pages that aren't meant to be cited.
- Pay per crawl sets a domain-level price on every crawl request. Pay per use pays out when content is actually used in an answer or a search result. Cloudflare announced experiments with Ceramic.ai and You.com in July 2026. Both are described as pre-launch.
- No. RSL is a standard for declaring license terms in a machine-readable form. Enforcement depends on crawlers choosing to comply or on separate blocking tools such as a CDN. Your declarations and your edge settings need to express the same intent.
- Most brand pages exist to be cited in AI answers, so free access is the right call. Monetization makes sense for pages where reading the source itself is the product, such as paid reports, databases, and news archives.
- We have not found public statistics linking licensing deals to citation frequency. Microsoft PCM, however, has announced a pilot that grounds Copilot responses in licensed content, so a setup where contracted content enters a given engine's grounding pool first already exists.
Q.If we turn on Pay per crawl, will our site disappear from AI answers?
Q.What is the difference between Pay per crawl and Pay per use?
Q.If we add RSL to robots.txt, will AI systems always follow it?
Q.Should brand or commerce sites consider monetizing their content?
Q.Do publishers with licensing deals get cited more in AI answers?
Sources
- [1] ↑Introducing pay per crawl — Cloudflare
- [2] ↑Making AI search smarter — Cloudflare
- [3] ↑Your site, your rules: new AI traffic options for all customers — Cloudflare
- [4] ↑The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals — Cloudflare
- [5] ↑RSL standard — RSL
- [6] ↑RSL AI Licensing 1.0 now an official industry standard with new capabilities as momentum accelerates — Editor and Publisher
- [7] ↑Building toward a sustainable content economy for the agentic web — Microsoft Advertising
- [8] ↑How Perplexity's new revenue model works, according to its head of publisher partnerships — Digiday
Related documents
- Cloudflare's Default AI Crawler Block, September 15, 2026: Will Your Site Vanish from AI Answers?A breakdown of the Search, Agent, and Training defaults Cloudflare put in force on September 15, 2026. Search crawlers stay allowed, while Agent blocks and mixed-purpose crawler inheritance are the real paths that cut citations in AI answers.
- Managing AI Crawlers: Allowing GPTBot, ClaudeBot, and PerplexityBot and the Trade-offsHow to identify GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended, and the visibility trade-offs of allowing or blocking them in robots.txt: based on OpenAI's and Google's official documentation.
- What Content Does AI Cite?: How Generative Engines Choose CitationsHow generative engines like ChatGPT and Perplexity pick the sources behind an answer, explained as a three-step process: retrieval, grounding, and synthesis, plus the conditions that make content citable: extractable chunks, semantic density, source credibility, and freshness.
- Google AI Mode and AI Overviews Publisher Opt-Out: The GEO Strategy Dilemma and Decision FrameworkGoogle rolled out AI Mode and AI Overviews opt-out toggles in Search Console in June 2026. Opting out eliminates GEO visibility entirely; staying in means AI consumes content for free while traffic erodes. Here is how the dilemma is structured and how different publisher types should choose.
- Korean Blog Platform AI Crawler Policies Compared 2026: Naver Blog, Tistory, and Brunch robots.txtNaver Blog blocks GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot outright in robots.txt. Tistory and velog carry no AI-crawler-specific rules, and Brunch blocks training bots only. Here is what each platform's raw robots.txt says and how that changes an article's odds of being cited in AI search.
- Multi-Engine Measurement: How to Measure Visibility Across ChatGPT, Gemini, Perplexity, and ClaudeWhy every engine answers differently, the trap of single-engine measurement, and a multi-engine GEO methodology for measuring AI visibility through prompt sets, repetition, and share of voice.
- What Is GEO: The Definition of Generative Engine Optimization and How It Differs From SEOGEO (Generative Engine Optimization) is the strategy of getting your content cited in answers produced by generative engines like ChatGPT and Perplexity. Here is the definition, how it differs from SEO, and how it works.
- Global GEO/AEO Player Landscape 2026, Monitoring Tools, Agencies, and PlatformsA 2026 landscape that sorts GEO/AEO players into monitoring tools, specialist solutions and agencies, enterprise platforms, and regional players. We compare the leading vendor in each category, founding, headquarters, tracked engines, pricing, and differentiation, against primary sources.