WWikiAP
Category: Methodology

How to Run a GEO A/B Test: Designing AI Citation Experiments with Control Groups, a Primary Metric, and Guardrails

A practical guide to designing GEO A/B tests that verify changes in AI citations. Compares page-split and single-page control designs, explains primary and guardrail metrics, walks through the public Omio and OnBoard cases, and ends with a six-step playbook and FAQ.

Content·AEO 에디터Published

Teams usually ask how to design a GEO A/B test right after they add a summary block or an FAQ and want to know whether AI citations went up. The most common mistake is to compare the month before the change with the month after and call it done. AI answers shift even when the question stays the same, and if a model update or a competitor's new content lands in the same window, a before-and-after comparison cannot tell you what caused the change. This guide covers two experiment designs that use a control group, how to separate the primary metric from guardrail metrics, three public cases (Omio, published by SearchPilot, and OnBoard and Bricx, covered in a single Peec AI article), and a six-step playbook you can use right away. If the metrics themselves are new to you, start with the GEO KPI and prompt monitoring guide.

30-second definitions: five terms used in GEO experiments

A GEO A/B test is an experiment that splits similar pages into a control group and a test group, applies a change only to the test group, and compares the difference in AI citations and LLM traffic.

A control group is a set of baseline pages left untouched for the whole test, which filters out outside factors such as model updates and seasonal shifts.

A primary metric is the single metric, chosen before the test starts, that decides success or failure.

A guardrail metric is a protective metric set in advance so that the change is not rolled out if this value gets worse, even when the primary metric improves.

A directional metric is a supporting metric that plays no part in the verdict and helps explain why the change happened.

Here is how the five fit together. The control group filters out outside factors, the primary metric delivers the verdict, guardrails block side effects, and directional metrics explain the reason. LeadGenLab gives an example setup: one primary metric, Google organic sessions and conversion rate as guardrails, and Search Console impressions, ChatGPT traffic, and mentions inside answers as directional metrics.[1]

GEO A/B test design flow 1 Hypothesis What change moves which result 2 Split Control and test from similar pages 3 Metrics One primary metric locked before start 4 Run Repeated measurement and checkpoints 5 Verdict Primary metric and guardrails Check guardrails first Stop if Google organic or conversion rate drops Roll out: primary up, guardrail OK Revise: primary up, guardrail down Discard: no change in primary
GEO A/B test design flow. Register metrics before the test starts, and check guardrails first when making the call (metric setup adapted from LeadGenLab 2026 and the SearchPilot case).

Comparing GEO experiment designs: page split, single-page control, before-and-after

GEO experiments differ in how they set up the control group. The table below puts the three designs seen in public cases, plus the common but not recommended before-and-after comparison, on the same criteria.

DesignControl setupPages neededMain metricsPublic caseLimitation
Page-split A/BA set of similar pages divided into two statistically similar groupsMany pages on the same templateLLM traffic, Google organic sessionsOmio (SearchPilot)Hard to apply on sites with few template pages
Single-page controlOne test page and one comparable page2 or moreCitation rate, retrieval count, visibilityOnBoard (Peec AI)The two pages start from different baselines, so read the gap with care
New-asset observationNo separate control; track a new domain or page1Whether it gets cited, retrieval share by engineBricx (Peec AI)Cannot establish causation; closer to hypothesis discovery
Before-and-afterNo control; the pre-change period is the baseline1 or moreAny metricNot recommendedCannot separate model updates and seasonal effects from the change itself

Commerce, travel, and real estate sites with large page counts should use a page split. SearchPilot's page-split method divides thousands of similar pages into statistically similar control and test groups, applies the GEO hypothesis to the test group only, and measures LLM-driven traffic such as ChatGPT alongside organic search traffic. The Omio test used this method; the number of pages involved was not disclosed.[3] For sites with only a few core pages, such as B2B SaaS, a single-page control is the realistic option.

Evidence: three public GEO experiments

Omio, page-split test (SearchPilot, n.d.). The first test added a module highlighting the brand's unique selling points (USPs) to each page. LLM sessions rose 18%, Google organic was inconclusive, and the change went live across all relevant pages.[3] The second test placed a bulleted summary with figures and comparison details above the existing summary table in the HTML. It helped LLM traffic but showed a likely 6.5% drop in Google organic sessions. Because Google organic is Omio's largest traffic source, the team held the rollout and decided to build a revised version.[3] LeadGenLab's takeaway from the case: AI traffic can rise while Google traffic falls, so both metrics have to be read together.[1]

OnBoard, single-page control test (Peec AI, 2026). OnBoard, a board management software company, added a Key Takeaways section made of standalone, citable sentences plus schema markup to the top of one page, and kept one comparable page as its control. The test ran for six weeks with checkpoints roughly every two weeks, and the team set citation rate as the primary metric before starting.[2]

MetricStartCheckpoint 1Checkpoint 2End
Citation rate, test page1.451.792.032.33
Citation rate, control page1.611.511.601.72
Retrievals, test page199222178166
Retrievals, control page3613617792
Visibility, test page22%12.4%7.3%9%

Source: (Peec AI, 2026)[2]

Over six weeks the test page's citation rate rose 61%, and absolute citations grew 34%, from 289 to 387. The control page's citation rate rose about 7% over the same period.[2] The row to watch is visibility: on the test page it fell from 22% to 9%. Without citation rate locked in as the primary metric beforehand, the team could have argued over the same data as either a win or a loss.

Bricx, new-domain observation (Peec AI, 2026). Bricx is the second team in the same Peec AI article as OnBoard.[2] Bricx, a UX design agency, bought a domain that nearly matched its target search term, built an agency directory on it, and tracked it for 67 days. With six referring domains and an authority score of 0, it was still cited within a few weeks on "best UX agency" style prompts. ChatGPT accounted for 91% of retrievals. On the exact head-keyword prompt, the site was retrieved in 38.6% of answers, but on prompts closer to purchase questions the rate was under 5%.[2] There was no control group, so read this as hypothesis discovery, not a causal finding.

Why repeated measurement is part of the design

AI answers vary from run to run, even for an identical question. SparkToro and Gumshoe ran 12 prompts 60 to 100 times each across ChatGPT, Claude, and Google AI. The chance that ChatGPT or Google AI returned the same list of brands in two responses was under 1 in 100, and the chance of the same list in the same order was roughly 1 in 1,000.[5] Before comparing test and control, you need to know how much each group's measurements move on their own.

Visibility margin of error by runs per prompt 5 runs 27pp 12 runs 12pp 100 runs 6pp Source: (Evertune, 2026)
Visibility margin of error by number of runs for a single prompt. Source: (Evertune, 2026)
RunsMargin of error (pp)Source
527pp(Evertune, 2026)
1212pp(Evertune, 2026)
1006pp(Evertune, 2026)

Evertune ran a single prompt repeatedly in ChatGPT and found a 27pp margin of error at 5 runs, shrinking to about 6pp at 100 runs.[6] Say the test group's citation rate rises 10pp while the measurement error is 27pp: that 10pp cannot support a verdict. For how to split effort between prompt count and repetitions, with formulas, see sample size standards for AI visibility scores. For why answers keep changing, see why AI brand recommendations change with every answer.

Metric design: one primary metric, guardrails, directional metrics

The usual mistake here is putting every metric you care about into the success criteria. With five metrics, one of them will improve by chance, and after the test ends the team reports success by picking whichever one went up.

TypeRoleExample metricsWhen to setBad practice
PrimaryDecides success or failureCitation rate in AI answers, or total organic sessions on the test pagesBefore the test startsSwitching to whichever metric improved after the test
GuardrailBlocks side effectsGoogle organic sessions, conversion rateBefore the test startsRolling out on LLM traffic alone with no guardrail
DirectionalExplains causesSearch Console impressions, ChatGPT traffic, mentions in answersBefore or during the testUsing a directional metric to make the call

LeadGenLab recommends keeping a single primary metric and notes that LLM traffic is often too small to produce a conclusive result.[1] On a site where LLM traffic runs in the dozens of visits per day, it is usually easier to reach a verdict by making the AI-answer citation rate the primary metric and demoting LLM traffic to a directional metric. The GA4 AI assistant channel measurement guide shows how to isolate LLM traffic in GA4. For rules on counting citations and mentions separately, see the difference between brand mentions and citations in AI answers.

Playbook: six steps for a GEO A/B test

StepWhat to doOutputCommon mistake
1 HypothesisWrite one sentence naming the single element you will change and the expected result"Adding a summary block raises the citation rate"Changing the summary, FAQ, and schema at once, so the cause cannot be isolated
2 SplitDivide same-template pages into two similar groups, or pick one pair of similar pagesURL lists for control and test groupsPacking only the high-traffic pages into the test group
3 Register metricsLock one primary metric, guardrails, directional metrics, and decision criteria in a documentExperiment design docChanging the primary metric mid-test
4 Prepare measurementChoose the prompt set, engines, and number of runs, then take baseline readings before the startBaseline tableShipping the change with no baseline
5 RunAt every checkpoint, measure both groups the same way and touch nothing except the test elementCheckpoint logMaking extra edits to the test group only
6 VerdictCheck guardrails first, then use the primary metric to roll out, revise, or discardVerdict record and next hypothesisRolling out despite a worse guardrail

If you serve the variant on a separate URL, follow Google's guidance. Cloaking, showing Googlebot a different URL from what users see, violates Google's spam policies even for testing. Google also says to point variant URLs to the original with rel=canonical, use a 302 rather than a 301 if a redirect is needed, and run the test only as long as necessary before removing the test elements.[4] A page-split design that changes HTML on the same URL avoids most of these issues.

The measurement tool in step 4 sets how precise your results can be. OnBoard and Bricx tracked citations and retrievals with Peec AI, and Bricx also used Google Search Console.[2] If you need to cover Korean-language prompts and Korean engines, a Korean measurement solution such as BOIDA is worth adding to the shortlist. Results differ widely by engine, so see the multi-engine measurement guide for logging each engine separately. To pick candidate changes to test, browse content structures AI likes to cite.

Key takeaways

Three things make a GEO A/B test result something you can actually judge: a control group that filters out outside factors, one primary metric locked before the start, and guardrails that keep you from trading Google traffic for AI traffic. The Omio case showed that even a GEO change that works has to be held back when it trips a guardrail. The OnBoard case showed that a pre-set primary metric lets you reach a verdict amid mixed signals such as a drop in visibility. Add repeated measurement to shrink the error, and you can tell whether a change is real or noise. For the broader concepts, start with what is GEO. Company details for LeadGenLab are on the LeadGenLab company page.

Related companies

Frequently asked questions

Q.How is a GEO A/B test different from an SEO A/B test?
Both use the same page-split method, but they watch different metrics. An SEO test tracks Google organic traffic; a GEO test tracks LLM traffic and citations inside AI answers. As the Omio case shows, one change can help LLM traffic and hurt Google organic, so you need to watch both (SearchPilot, n.d.).
Q.Can I run a GEO experiment if my site has only a few pages?
Yes, but the design changes. With hundreds of similar pages or more, a page-split test that divides pages into groups fits. With only a handful, pair one test page with one comparison page and track both over the same period. OnBoard took the second route, comparing citation rates over six weeks with checkpoints every two weeks (Peec AI, 2026).
Q.Should the primary metric be LLM traffic or citation rate?
Let data volume decide. LLM traffic sits closer to revenue, but its volume is small and often produces inconclusive results (LeadGenLab, 2026). On low-traffic sites, making the AI-answer citation rate the primary metric and treating LLM traffic as a directional metric gives you a better chance of reaching a verdict.
Q.How long should the experiment run?
There is no fixed rule. Run it until the primary metric you set in advance reaches a level you can judge. Google does recommend running a test only as long as necessary and removing the test elements as soon as it ends (Google Search Central, n.d.). Among the public cases, OnBoard ran for six weeks (Peec AI, 2026).
Q.Can I put the variant page on a separate URL during the test?
If you do, point the variant URL to the original with rel=canonical, and use a 302 redirect rather than a 301 if you need a redirect. Cloaking, showing Googlebot different content from what users see, violates Google's spam policies even for testing (Google Search Central, n.d.).
Q.If the test page's visibility fell but its citation rate rose, is that a success?
Judge it by the primary metric you set beforehand. In the OnBoard test, the test page's citation rate climbed from 1.45 to 2.33 while its visibility fell from 22% to 9% (Peec AI, 2026). If citation rate was the primary metric, call it a success, and log the visibility drop as a hypothesis for the next experiment.

Sources

  1. [1] ↑GEO A/B 테스트, AI 유입과 구글 유입을 함께 보는 실험 설계법 — 리드젠랩
  2. [2] ↑How two teams run their own GEO experiments — Peec AI
  3. [3] ↑GEO A/B Testing: SEO and GEO Are Not The Same — SearchPilot
  4. [4] ↑Minimize A/B testing impact in Google Search — Google Search Central
  5. [5] ↑AIs are highly inconsistent when recommending brands or products — SparkToro
  6. [6] ↑How many repetitions of a single prompt does it take to understand your AI visibility? — Evertune

This document was last edited on Nov 11, 2026. WikiAP content is compiled from public primary sources and updated for accuracy.