How to Run a GEO A/B Test: Designing AI Citation Experiments with Control Groups, a Primary Metric, and Guardrails
A practical guide to designing GEO A/B tests that verify changes in AI citations. Compares page-split and single-page control designs, explains primary and guardrail metrics, walks through the public Omio and OnBoard cases, and ends with a six-step playbook and FAQ.
Teams usually ask how to design a GEO A/B test right after they add a summary block or an FAQ and want to know whether AI citations went up. The most common mistake is to compare the month before the change with the month after and call it done. AI answers shift even when the question stays the same, and if a model update or a competitor's new content lands in the same window, a before-and-after comparison cannot tell you what caused the change. This guide covers two experiment designs that use a control group, how to separate the primary metric from guardrail metrics, three public cases (Omio, published by SearchPilot, and OnBoard and Bricx, covered in a single Peec AI article), and a six-step playbook you can use right away. If the metrics themselves are new to you, start with the GEO KPI and prompt monitoring guide.
30-second definitions: five terms used in GEO experiments
A GEO A/B test is an experiment that splits similar pages into a control group and a test group, applies a change only to the test group, and compares the difference in AI citations and LLM traffic.
A control group is a set of baseline pages left untouched for the whole test, which filters out outside factors such as model updates and seasonal shifts.
A primary metric is the single metric, chosen before the test starts, that decides success or failure.
A guardrail metric is a protective metric set in advance so that the change is not rolled out if this value gets worse, even when the primary metric improves.
A directional metric is a supporting metric that plays no part in the verdict and helps explain why the change happened.
Here is how the five fit together. The control group filters out outside factors, the primary metric delivers the verdict, guardrails block side effects, and directional metrics explain the reason. LeadGenLab gives an example setup: one primary metric, Google organic sessions and conversion rate as guardrails, and Search Console impressions, ChatGPT traffic, and mentions inside answers as directional metrics.[1]
Comparing GEO experiment designs: page split, single-page control, before-and-after
GEO experiments differ in how they set up the control group. The table below puts the three designs seen in public cases, plus the common but not recommended before-and-after comparison, on the same criteria.
| Design | Control setup | Pages needed | Main metrics | Public case | Limitation |
|---|---|---|---|---|---|
| Page-split A/B | A set of similar pages divided into two statistically similar groups | Many pages on the same template | LLM traffic, Google organic sessions | Omio (SearchPilot) | Hard to apply on sites with few template pages |
| Single-page control | One test page and one comparable page | 2 or more | Citation rate, retrieval count, visibility | OnBoard (Peec AI) | The two pages start from different baselines, so read the gap with care |
| New-asset observation | No separate control; track a new domain or page | 1 | Whether it gets cited, retrieval share by engine | Bricx (Peec AI) | Cannot establish causation; closer to hypothesis discovery |
| Before-and-after | No control; the pre-change period is the baseline | 1 or more | Any metric | Not recommended | Cannot separate model updates and seasonal effects from the change itself |
Commerce, travel, and real estate sites with large page counts should use a page split. SearchPilot's page-split method divides thousands of similar pages into statistically similar control and test groups, applies the GEO hypothesis to the test group only, and measures LLM-driven traffic such as ChatGPT alongside organic search traffic. The Omio test used this method; the number of pages involved was not disclosed.[3] For sites with only a few core pages, such as B2B SaaS, a single-page control is the realistic option.
Evidence: three public GEO experiments
Omio, page-split test (SearchPilot, n.d.). The first test added a module highlighting the brand's unique selling points (USPs) to each page. LLM sessions rose 18%, Google organic was inconclusive, and the change went live across all relevant pages.[3] The second test placed a bulleted summary with figures and comparison details above the existing summary table in the HTML. It helped LLM traffic but showed a likely 6.5% drop in Google organic sessions. Because Google organic is Omio's largest traffic source, the team held the rollout and decided to build a revised version.[3] LeadGenLab's takeaway from the case: AI traffic can rise while Google traffic falls, so both metrics have to be read together.[1]
OnBoard, single-page control test (Peec AI, 2026). OnBoard, a board management software company, added a Key Takeaways section made of standalone, citable sentences plus schema markup to the top of one page, and kept one comparable page as its control. The test ran for six weeks with checkpoints roughly every two weeks, and the team set citation rate as the primary metric before starting.[2]
| Metric | Start | Checkpoint 1 | Checkpoint 2 | End |
|---|---|---|---|---|
| Citation rate, test page | 1.45 | 1.79 | 2.03 | 2.33 |
| Citation rate, control page | 1.61 | 1.51 | 1.60 | 1.72 |
| Retrievals, test page | 199 | 222 | 178 | 166 |
| Retrievals, control page | 36 | 136 | 177 | 92 |
| Visibility, test page | 22% | 12.4% | 7.3% | 9% |
Source: (Peec AI, 2026)[2]
Over six weeks the test page's citation rate rose 61%, and absolute citations grew 34%, from 289 to 387. The control page's citation rate rose about 7% over the same period.[2] The row to watch is visibility: on the test page it fell from 22% to 9%. Without citation rate locked in as the primary metric beforehand, the team could have argued over the same data as either a win or a loss.
Bricx, new-domain observation (Peec AI, 2026). Bricx is the second team in the same Peec AI article as OnBoard.[2] Bricx, a UX design agency, bought a domain that nearly matched its target search term, built an agency directory on it, and tracked it for 67 days. With six referring domains and an authority score of 0, it was still cited within a few weeks on "best UX agency" style prompts. ChatGPT accounted for 91% of retrievals. On the exact head-keyword prompt, the site was retrieved in 38.6% of answers, but on prompts closer to purchase questions the rate was under 5%.[2] There was no control group, so read this as hypothesis discovery, not a causal finding.
Why repeated measurement is part of the design
AI answers vary from run to run, even for an identical question. SparkToro and Gumshoe ran 12 prompts 60 to 100 times each across ChatGPT, Claude, and Google AI. The chance that ChatGPT or Google AI returned the same list of brands in two responses was under 1 in 100, and the chance of the same list in the same order was roughly 1 in 1,000.[5] Before comparing test and control, you need to know how much each group's measurements move on their own.
| Runs | Margin of error (pp) | Source |
|---|---|---|
| 5 | 27pp | (Evertune, 2026) |
| 12 | 12pp | (Evertune, 2026) |
| 100 | 6pp | (Evertune, 2026) |
Evertune ran a single prompt repeatedly in ChatGPT and found a 27pp margin of error at 5 runs, shrinking to about 6pp at 100 runs.[6] Say the test group's citation rate rises 10pp while the measurement error is 27pp: that 10pp cannot support a verdict. For how to split effort between prompt count and repetitions, with formulas, see sample size standards for AI visibility scores. For why answers keep changing, see why AI brand recommendations change with every answer.
Metric design: one primary metric, guardrails, directional metrics
The usual mistake here is putting every metric you care about into the success criteria. With five metrics, one of them will improve by chance, and after the test ends the team reports success by picking whichever one went up.
| Type | Role | Example metrics | When to set | Bad practice |
|---|---|---|---|---|
| Primary | Decides success or failure | Citation rate in AI answers, or total organic sessions on the test pages | Before the test starts | Switching to whichever metric improved after the test |
| Guardrail | Blocks side effects | Google organic sessions, conversion rate | Before the test starts | Rolling out on LLM traffic alone with no guardrail |
| Directional | Explains causes | Search Console impressions, ChatGPT traffic, mentions in answers | Before or during the test | Using a directional metric to make the call |
LeadGenLab recommends keeping a single primary metric and notes that LLM traffic is often too small to produce a conclusive result.[1] On a site where LLM traffic runs in the dozens of visits per day, it is usually easier to reach a verdict by making the AI-answer citation rate the primary metric and demoting LLM traffic to a directional metric. The GA4 AI assistant channel measurement guide shows how to isolate LLM traffic in GA4. For rules on counting citations and mentions separately, see the difference between brand mentions and citations in AI answers.
Playbook: six steps for a GEO A/B test
| Step | What to do | Output | Common mistake |
|---|---|---|---|
| 1 Hypothesis | Write one sentence naming the single element you will change and the expected result | "Adding a summary block raises the citation rate" | Changing the summary, FAQ, and schema at once, so the cause cannot be isolated |
| 2 Split | Divide same-template pages into two similar groups, or pick one pair of similar pages | URL lists for control and test groups | Packing only the high-traffic pages into the test group |
| 3 Register metrics | Lock one primary metric, guardrails, directional metrics, and decision criteria in a document | Experiment design doc | Changing the primary metric mid-test |
| 4 Prepare measurement | Choose the prompt set, engines, and number of runs, then take baseline readings before the start | Baseline table | Shipping the change with no baseline |
| 5 Run | At every checkpoint, measure both groups the same way and touch nothing except the test element | Checkpoint log | Making extra edits to the test group only |
| 6 Verdict | Check guardrails first, then use the primary metric to roll out, revise, or discard | Verdict record and next hypothesis | Rolling out despite a worse guardrail |
If you serve the variant on a separate URL, follow Google's guidance. Cloaking, showing Googlebot a different URL from what users see, violates Google's spam policies even for testing. Google also says to point variant URLs to the original with rel=canonical, use a 302 rather than a 301 if a redirect is needed, and run the test only as long as necessary before removing the test elements.[4] A page-split design that changes HTML on the same URL avoids most of these issues.
The measurement tool in step 4 sets how precise your results can be. OnBoard and Bricx tracked citations and retrievals with Peec AI, and Bricx also used Google Search Console.[2] If you need to cover Korean-language prompts and Korean engines, a Korean measurement solution such as BOIDA is worth adding to the shortlist. Results differ widely by engine, so see the multi-engine measurement guide for logging each engine separately. To pick candidate changes to test, browse content structures AI likes to cite.
Key takeaways
Three things make a GEO A/B test result something you can actually judge: a control group that filters out outside factors, one primary metric locked before the start, and guardrails that keep you from trading Google traffic for AI traffic. The Omio case showed that even a GEO change that works has to be held back when it trips a guardrail. The OnBoard case showed that a pre-set primary metric lets you reach a verdict amid mixed signals such as a drop in visibility. Add repeated measurement to shrink the error, and you can tell whether a change is real or noise. For the broader concepts, start with what is GEO. Company details for LeadGenLab are on the LeadGenLab company page.
Related companies
- 리드젠랩 (LeadGenLab)AI 가시성 최적화 에이전시
- 보이다 (BOIDA)생성형 검색 최적화(GEO) 솔루션, AI 가시성 측정
- Peec AIAI 가시성 모니터링 플랫폼
Frequently asked questions
- Both use the same page-split method, but they watch different metrics. An SEO test tracks Google organic traffic; a GEO test tracks LLM traffic and citations inside AI answers. As the Omio case shows, one change can help LLM traffic and hurt Google organic, so you need to watch both (SearchPilot, n.d.).
- Yes, but the design changes. With hundreds of similar pages or more, a page-split test that divides pages into groups fits. With only a handful, pair one test page with one comparison page and track both over the same period. OnBoard took the second route, comparing citation rates over six weeks with checkpoints every two weeks (Peec AI, 2026).
- Let data volume decide. LLM traffic sits closer to revenue, but its volume is small and often produces inconclusive results (LeadGenLab, 2026). On low-traffic sites, making the AI-answer citation rate the primary metric and treating LLM traffic as a directional metric gives you a better chance of reaching a verdict.
- There is no fixed rule. Run it until the primary metric you set in advance reaches a level you can judge. Google does recommend running a test only as long as necessary and removing the test elements as soon as it ends (Google Search Central, n.d.). Among the public cases, OnBoard ran for six weeks (Peec AI, 2026).
- If you do, point the variant URL to the original with rel=canonical, and use a 302 redirect rather than a 301 if you need a redirect. Cloaking, showing Googlebot different content from what users see, violates Google's spam policies even for testing (Google Search Central, n.d.).
- Judge it by the primary metric you set beforehand. In the OnBoard test, the test page's citation rate climbed from 1.45 to 2.33 while its visibility fell from 22% to 9% (Peec AI, 2026). If citation rate was the primary metric, call it a success, and log the visibility drop as a hypothesis for the next experiment.
Q.How is a GEO A/B test different from an SEO A/B test?
Q.Can I run a GEO experiment if my site has only a few pages?
Q.Should the primary metric be LLM traffic or citation rate?
Q.How long should the experiment run?
Q.Can I put the variant page on a separate URL during the test?
Q.If the test page's visibility fell but its citation rate rose, is that a success?
Sources
- [1] ↑GEO A/B 테스트, AI 유입과 구글 유입을 함께 보는 실험 설계법 — 리드젠랩
- [2] ↑How two teams run their own GEO experiments — Peec AI
- [3] ↑GEO A/B Testing: SEO and GEO Are Not The Same — SearchPilot
- [4] ↑Minimize A/B testing impact in Google Search — Google Search Central
- [5] ↑AIs are highly inconsistent when recommending brands or products — SparkToro
- [6] ↑How many repetitions of a single prompt does it take to understand your AI visibility? — Evertune
Related documents
- How Many Prompts and Runs Make an AI Visibility Score Trustworthy? Sample Size and Repeat-Run GuidelinesHow many prompts, run how many times, before you can trust an AI visibility score. Covers confidence intervals by response count, the responses needed to compare two periods, the design effect of repeated runs, public experiments from SparkToro, Evertune and others, and a five-step sampling procedure.
- GEO KPI Performance Measurement: Citation Rate, AI SoV, and Prompt MonitoringA step-by-step guide to the core GEO KPIs: citation rate, AI Share of Voice, and prompt monitoring. Covers the arXiv-grounded statistical framework, brand-scale benchmarks, and a side-by-side comparison of domestic and global measurement tools for quantifying AI search visibility.
- GA4 AI Assistant Channel: Measuring ChatGPT, Gemini, and Claude TrafficGA4's AI Assistant channel: added to Default Channel Groups in May 2026, automatically classifies AI referral traffic, yet a structural blind spot remains: sessions that arrive without a referrer header. Here's what the channel captures, what it misses, and how to cover the gap.
- Why AI Recommends Different Brands Every Time You Ask the Same Question: 5 Causes of Response Variability and How to Measure Around ThemAsk ChatGPT, Claude or Google AI the same question twice and the recommended brands change. This guide splits the cause into five layers (sampling, inference nondeterminism, query fan-out search, personalization and prompt wording), backs each with public experiments from SparkToro and others, and lays out a five-step measurement response.
- Content Structure That Gets Cited in AI Answers: Writing for ExtractabilityThe writing AI cites is not the same as writing that reads well. How to raise extractability through citable units, answer-first placement, question-answer structure, and tables, lists, and definitions: grounded in GEO research and a practical checklist.
- Brand Mentions vs. Citations in AI Answers: What to Measure, Ghost Citations, and Four MetricsA brand mention (your name in the answer) and a citation (your link as a source) are different signals in AI answers. Using measured data, including a 61.7% ghost citation share and wide link-rate gaps between engines, this guide shows how to track mention rate, citation rate, and linked mention rate separately by engine and query type.
- Multi-Engine Measurement: How to Measure Visibility Across ChatGPT, Gemini, Perplexity, and ClaudeWhy every engine answers differently, the trap of single-engine measurement, and a multi-engine GEO methodology for measuring AI visibility through prompt sets, repetition, and share of voice.
- What Is GEO: The Definition of Generative Engine Optimization and How It Differs From SEOGEO (Generative Engine Optimization) is the strategy of getting your content cited in answers produced by generative engines like ChatGPT and Perplexity. Here is the definition, how it differs from SEO, and how it works.