How Many Prompts and Runs Make an AI Visibility Score Trustworthy? Sample Size and Repeat-Run Guidelines
How many prompts, run how many times, before you can trust an AI visibility score. Covers confidence intervals by response count, the responses needed to compare two periods, the design effect of repeated runs, public experiments from SparkToro, Evertune and others, and a five-step sampling procedure.
Teams that adopt an AI visibility tool almost always ask the same thing in month two: how many prompts does it take before the score means anything? Month one says 32%, month two says 27%, and someone has to write the report without knowing whether that 5pp drop is real or just sampling error. The answer lives in the response count. A visibility score is the share of AI responses that mention the brand, and the precision of any proportion comes down to how many responses you have and how independent they are. This guide covers how to work backward from a target precision to the prompts and repeat runs you need, why repeated responses add less precision than they seem to, what the published repeat-run experiments found, and a five-step procedure for locking the measurement design into a written spec.
30-second definitions: five sampling terms
Response sample size (n) is the total number of AI responses that feed a visibility score, equal to prompts times repeat runs times engines.
Repeated runs are a collection method that sends the same prompt to the same engine several times and averages out response-to-response variation.
A confidence interval is the range around the observed mention rate that is expected to contain the true mention rate, and it narrows as responses increase.
The design effect is the factor by which the variance of a correlated sample, such as repeated responses, exceeds that of an independent sample, calculated as 1+(m-1)ρ.
Effective sample size is the number of independent responses that carry the same information as n correlated responses, found by dividing n by the design effect.
The five terms form a single chain. Multiply prompts, repeats, and engines to get the response count; divide out the correlation that repeats introduce, via the design effect, to get the effective sample; then use the effective sample to calculate the width of the confidence interval. According to Wikipedia's 'Design effect' entry, Leslie Kish defined the design effect and effective sample size in his 1965 book Survey Sampling.[7]
How many responses buy how many points of precision
A visibility score is a binomial proportion. If p is the share of n responses that mention the brand, the normal approximation gives a 95% confidence interval half-width of 1.96×√(p(1-p)/n). The NIST/SEMATECH handbook warns that this approximation can be inaccurate when the sample is small or the proportion is near 0 or 1, and it recommends the Wilson method for almost every combination of n and p.[5] The chart and table below use normal-approximation values under an assumption of independent responses. Treat them as order-of-magnitude guides.
| Response sample size (n) | 95% CI half-width (normal approx.) | Wilson 95% CI | Gap you can call |
|---|---|---|---|
| 30 | about ±16.4pp | about 16.7% to 47.9% | Only gaps of 15pp or more |
| 50 | about ±12.7pp | Not calculated | High double-digit pp gaps |
| 100 | about ±9.0pp | about 21.9% to 39.6% | Double-digit pp gaps |
| 300 | about ±5.2pp | about 25.1% to 35.4% | High single-digit pp gaps |
| 1,000 | about ±2.8pp | Not calculated | Gaps of a few pp |
Every value assumes an observed mention rate of 30% and was calculated directly from the normal-approximation and Wilson formulas in the NIST/SEMATECH handbook.[5] At 30 responses the normal-approximation interval runs from 13.6% to 46.4%, while the Wilson interval runs from 16.7% to 47.9%, which is asymmetric. The gap between the two methods widens as the sample shrinks, so Wilson values are the safer choice for reports.
Run the formula in reverse and you get the responses required for a target precision. Plugging in the worst case, a 50% mention rate, gives about 97 responses for a 10pp half-width, about 385 for 5pp, and about 1,068 for 3pp. The single-prompt benchmarks from Gumshoe that cloro cites (97 runs for ±10pp, 385 runs for ±5pp) come from the same formula.[8]
Comparing two periods takes more responses
Reading a score at one point in time and deciding whether two points differ are separate problems. A comparison carries sampling error on both sides, so it needs more responses. The NIST/SEMATECH handbook gives a formula for the sample size of a proportion test based on the significance level (α), power (1-β), and the difference to detect (δ).[6]
| Comparison | Responses needed per period (independent responses) | Conditions |
|---|---|---|
| Mention rate 20% vs 30% (10pp gap) | about 294 | Two-sided α 5%, power 80% |
| Mention rate 30% vs 35% (5pp gap) | about 1,377 | Two-sided α 5%, power 80% |
| Single point, half-width ±5pp (assumes 50% mention rate) | about 385 | 95% confidence level |
| Single point, half-width ±3pp (assumes 50% mention rate) | about 1,068 | 95% confidence level |
The figure of about 294 responses per period to separate 20% from 30% matches cloro's public guide.[8] The 5pp swings that show up all the time in practice can only be called with more than 1,000 independent responses per period. If the design runs 50 prompts once a month on one engine, a 5pp move in the monthly score gives no grounds for a performance claim.
Repeating the same prompt adds less precision
The tables above assume independent responses, and AI visibility measurement breaks that assumption. Run one prompt 100 times and the wording of each response changes, but whatever makes that prompt favorable or unfavorable to the brand holds across all 100 runs. Statistics treats this as a cluster sample: with cluster size m and within-cluster correlation ρ, the design effect is 1+(m-1)ρ. The effective sample size is the total sample divided by the design effect.[7]
| Structure of 1,000 responses (hypothetical) | Design effect (assumes ρ=0.3) | Effective sample size | Half-width at 30% mention rate |
|---|---|---|---|
| 1,000 prompts x 1 run | 1.0 | 1,000 | about ±2.8pp |
| 500 prompts x 2 runs | 1.3 | about 769 | about ±3.2pp |
| 100 prompts x 10 runs | 3.7 | about 270 | about ±5.5pp |
| 20 prompts x 50 runs | 15.7 | about 64 | about ±11.3pp |
| 10 prompts x 100 runs | 30.7 | about 33 | about ±15.7pp |
ρ=0.3 is a hypothetical value chosen to show how the formula behaves; it is not a measured figure. Actual ρ varies by category and engine and has to be estimated from your own data. The direction is clear regardless. Hold total responses constant, shift toward more repeats and fewer prompts, and the effective sample falls fast. That is why 1,000 responses on their own do not justify expecting ±3pp precision.
None of this makes repeats useless. They are what you need to pin down the mention rate of one specific prompt. For brand-level visibility, the number of distinct prompts sets the precision. Prompt selection is covered in prompt volume vs search volume.
What the published repeat-run experiments show
The public experiments on repeat counts answer different questions. Read a single-prompt diagnostic study next to a large portfolio study without separating them and the conclusions appear to conflict.
| Source | Design | Key result | How to read it |
|---|---|---|---|
| SparkToro (2026) | 600 volunteers, 12 prompts, 2,961 total runs across ChatGPT, Claude, and Google AI responses, collected November to December 2025 | Asking ChatGPT or Google AI 100 times gave under a 1% chance of the same brand list, and the same order came up about once in 1,000 runs | Rank is unusable as a metric; mention rate averaged over repeats is usable[1] |
| Evertune (2026) | 10,700 ChatGPT prompt runs, measuring Acura's mention rate on a single SUV prompt | At a mention rate of about 10%, the margin of error was 27pp at 5 runs and about 6pp at 100 runs, with diminishing returns past 100 | Repeat benchmark for single-prompt diagnosis[2] |
| Meikai (2026) | 500 US English prompts, 5 engines, 10 days, 250,000 responses, comparing 1 to 10 runs per day | 10-day aggregate citation share of 20.52%; 1 and 10 runs per day gave essentially the same result. P95 error 0.17pp at 1 run, 0.05pp at 10 runs | In a large portfolio, prompt count sets precision[3] |
| Citation Labs (2026) | One B2B company, 4,579 purchase-intent prompts, 4 engines, 4 runs over 12 weeks | Order and cited URLs changed between runs, but appearance rate stayed stable within the project. Mention rate by engine: Google AI Overviews 25.33%, Google AI Mode 83.16% | Report each engine separately[4] |
In this study SparkToro wrote that prompts generally need to be run at least 60 to 100 times and averaged, and it judged visibility percentages from dozens to hundreds of prompts run multiple times to be a reasonable metric. It also wrote that tools reporting rank position inside AI answers are hard to trust.[1] Evertune's result means that at 5 runs, anything from 0% to 37% is plausible.[2] A single-prompt score built on 5 runs cannot support a claim about the gap to a competitor.
The Meikai experiment had one exception. Copilot's citation-share error was 1.33pp at one run per day and 0.49pp at ten, and the first policy to drop below 1pp was three runs a day. The author noted that the results are limited to this 500-prompt portfolio and 10-day window, and that smaller portfolios, single-prompt diagnosis, or intraday measurement may need more repeats.[3] Citation Labs also offered an operating rule: if more than 10% of prompts change, treat the set as a new cohort.[4] Collection methods per engine are laid out in multi-engine measurement methodology.
Cause, impact, action
Cause. AI responses vary from run to run, and a visibility score is the proportion across those responses. With a small sample, the measured value can land far from the true mention rate.
Impact. Reporting month-over-month changes from scores built on fewer than 100 responses mistakes sampling error for performance change. A sample built mostly from repeats delivers less precision than its response count suggests. Layer on the problem of different tools giving different scores and interpretation gets harder still; that issue is covered in why AI visibility tools report different scores.
Action. Set the target precision first, back out the effective sample it requires, then split that budget between prompts and repeats. The five steps below lay out the procedure.
Five steps to a sampling design
| Step | What to do | Output | Pass criterion |
|---|---|---|---|
| 1. Define the decision target | Set the smallest gap the report must distinguish (e.g., 10pp vs a competitor, 5pp month over month) | Target gap and confidence level | Is the gap written down as a number? |
| 2. Back out required responses | Use the proportion confidence interval formula for a single point and the two-proportion test formula for comparing two points | Required effective sample by engine and by period | Are the conditions (α, power, baseline mention rate) recorded alongside? |
| 3. Allocate prompts and repeats | Add distinct prompts first; add repeats only for single-prompt diagnosis or high-variance engines | Prompt list, repeat policy | Does the effective sample, after the design effect, exceed the step 2 value? |
| 4. Freeze the spec | Document engine and model, region, language, login state, collection cadence, and prompt version | Measurement spec | Are the share of changed prompts and the timing of changes recorded? |
| 5. Report with intervals | Attach the response count and Wilson 95% interval to every score; do not report changes with overlapping intervals as performance | Reporting template | Does every number have n and an interval next to it? |
Step 1 is the one most often skipped. Without a target gap there is no way to set the required sample, and the tool's default ends up deciding the sample size. To tell executives "we are ahead of the competitor," for instance, the difference between the two brands' scores has to exceed the interval width.
In step 3, prompt diversity comes first. Variants that only reword the same question are highly correlated with each other and behave much like repeats. Splitting prompts by purchase stage, persona, and comparison axis so they carry different intents grows the effective sample. Prompt panel design continues in GEO KPI and performance measurement.
Step 4 keeps the trend line continuous. When a model is swapped or a large share of prompts is replaced, break the trend line and annotate it. Whatever measurement tool you use, whether Profound, Peec AI, or Korea's BOIDA, get the prompt count, repeat policy, engine list, and collection cadence in writing and copy them into the spec. A feature comparison of the tools is in AI visibility monitoring tools compared.
Step 5 lines up with the industry standards discussion. In a framework published in August 2026, the IAB split AI visibility measurement into Decision-grade and Directional, and addressed stability, reproducibility, and disclosure requirements for measurement providers.[9] A score without a response count and interval is best read as directional. The framework is explained in the IAB Four Ps measurement standard.
Three common small-sample mistakes
The first is reporting zero mentions as 0% visibility. With zero mentions in 12 responses, the upper bound of the Wilson 95% interval is about 24.2% (calculated with the Wilson method recommended by NIST/SEMATECH).[5] Twelve responses cannot support the conclusion that "AI never mentions our brand."
The second is pooling engines into one score. Even on the same prompt set, there is a case where mention rates split into the 25% range on one engine and the 83% range on another.[4] If the engine mix changes, the pooled score moves with no connection to brand performance. Per-engine share formulas are in how to measure AI search share of voice.
The third is using average rank as a metric. If the same ordered list comes up about once in 1,000 runs, an average rank is built from values drawn from a different list every time.[1] Mention rate, which only counts whether the brand appears, holds up better under repeated measurement.
Summary
Whether an AI visibility score can be trusted depends on the effective sample size, not the raw response count. At an observed mention rate of 30%, 300 independent responses give a half-width of about 5.2pp and 1,000 give about 2.8pp, while repeated responses to the same prompt contribute less precision in proportion to the design effect. For single-prompt diagnosis, 60 to 100 runs is the baseline from public experiments; for brand-level visibility, adding distinct prompts comes first.
The reporting rule fits in one line: put the response count and interval next to every score, and do not call it a change when the intervals overlap. For the definition and scope of GEO, see what is GEO.
Related companies
- 보이다 (BOIDA)생성형 검색 최적화(GEO) 솔루션, AI 가시성 측정
- Peec AIAI 가시성 모니터링 플랫폼
- ProfoundAI 가시성 모니터링 플랫폼
Frequently asked questions
- There is no fixed minimum; you work backward from the precision you need. Under the worst case (a 50% mention rate), a 95% confidence interval half-width of 10pp takes about 97 independent responses, 5pp takes about 385, and 3pp takes about 1,068 (calculated with the NIST/SEMATECH proportion interval formula). Repeated responses to one prompt are correlated, so it is safer to hit these numbers with distinct prompts rather than with repeat runs.
- It depends on the job. Diagnosing the mention rate of a single prompt needs repeats: SparkToro recommends averaging at least 60 to 100 runs (SparkToro, 2026), and Evertune found a 27pp margin of error at 5 runs versus about 6pp at 100 (Evertune, 2026). Across a 500-prompt portfolio, though, one run a day and ten runs a day gave essentially identical 10-day aggregate results (Meikai, 2026).
- No. Twenty prompts run 50 times each and 1,000 prompts run once each both yield 1,000 responses, but they carry different amounts of information. When responses within a prompt resemble each other (intraclass correlation ρ), the effective sample equals the total response count divided by the design effect 1+(m-1)ρ (Wikipedia, 'Design effect', accessed 2026). With ρ at 0.3, the 20 x 50 setup has an effective sample of about 64.
- Check the sample size first. Distinguishing a move from 30% to 35% at 5% significance and 80% power takes about 1,377 independent responses per period (calculated with the two-proportion formula). With only a few hundred responses per period, a 5pp change can sit inside sampling error, and it should not be reported as a performance change.
- The observed value is 0%, but that does not mean the true mention rate is 0%. With zero mentions in 12 responses, the upper bound of the Wilson 95% confidence interval is about 24.2% (calculated with the Wilson method recommended by NIST/SEMATECH). The smaller the sample, the weaker a zero result is as evidence, so collect more responses before drawing a conclusion.
- Look at each engine's score separately before pooling. For the same client and the same prompt set, brand mention rates were 25.33% in Google AI Overviews and 83.16% in Google AI Mode (Citation Labs, 2026). When engines diverge that much, a pooled score moves with the engine mix, so report sample size and confidence interval for each engine.
Q.What is the minimum number of prompts needed to measure AI visibility?
Q.How many times should the same prompt be repeated?
Q.Do 1,000 responses give the same precision regardless of how the prompts are structured?
Q.Our score rose 5pp over last month. Does that count as an improvement?
Q.If the brand was never mentioned, is visibility 0%?
Q.With several engines, can I pool the responses into one score?
Sources
- [1] ↑AIs are highly inconsistent when recommending brands or products — SparkToro
- [2] ↑How many repetitions of a single prompt does it take to understand your AI visibility? — Evertune
- [3] ↑How many times should you run an AI visibility prompt? — Meikai
- [4] ↑Yes, AI Rank Is Trackable: Results From 4,579 Buyer Prompts — Citation Labs
- [5] ↑7.2.4.1. Confidence intervals (proportion) — NIST/SEMATECH e-Handbook of Statistical Methods
- [6] ↑7.2.4.2. Sample sizes required — NIST/SEMATECH e-Handbook of Statistical Methods
- [7] ↑Design effect — Wikipedia
- [8] ↑AI Visibility Sample Size: How Many Prompts and Runs You Need — cloro
- [9] ↑Measuring Visibility in the AI Era — IAB
Related documents
- Why AI Visibility Tools Report Different Scores for the Same Brand: Causes and a Verification ProcedureWhy the same brand scores differently in every AI visibility tool, split into four layers (engine variance, collection design, metric definition, aggregation), with a side-by-side table of the published formulas from Profound, Peec AI, and Ahrefs, sample-size confidence interval thresholds, and a six-step verification procedure.
- GEO KPI Performance Measurement: Citation Rate, AI SoV, and Prompt MonitoringA step-by-step guide to the core GEO KPIs: citation rate, AI Share of Voice, and prompt monitoring. Covers the arXiv-grounded statistical framework, brand-scale benchmarks, and a side-by-side comparison of domestic and global measurement tools for quantifying AI search visibility.
- Multi-Engine Measurement: How to Measure Visibility Across ChatGPT, Gemini, Perplexity, and ClaudeWhy every engine answers differently, the trap of single-engine measurement, and a multi-engine GEO methodology for measuring AI visibility through prompt sets, repetition, and share of voice.
- What Is AI Search Share of Voice: Definition, Measurement Formula, and Brand Visibility GuideAI Search Share of Voice (AI SOV) is the percentage of AI-generated answers from ChatGPT, Perplexity, and Gemini that mention a specific brand. This page covers the formula, how it differs from traditional SOV, per-engine measurement methods, and a tool comparison: all in one place.
- IAB AI Visibility Measurement Standard: The Four Ps Framework ExplainedThe IAB's August 2026 AI visibility measurement standard defines Four Ps: Presence, Prominence, Portrayal, and Persuasion, along with two data quality tiers and vendor disclosure requirements. This page covers the full framework, an industry concentration breakdown, a tool comparison, and a brand execution roadmap.
- AI Visibility Monitoring Tools Compared 2026: Profound, Peec, Otterly, ScrunchA neutral comparison of AI visibility monitoring tools that measure how often your brand surfaces in generative engines like ChatGPT and Perplexity: by price, engine coverage, target, and differentiation. Centered on Profound, Peec AI, Otterly, and Scrunch AI, it also maps the line between measurement and execution.
- Prompt Volume vs. Search Volume: How Far to Trust GEO Tool Numbers, How They Are Calculated, and How to Verify ThemPrompt volume and keyword search volume differ in source, unit, and method. This guide compares the published methods of Profound, Semrush, and Similarweb, lays out a 5-step process for using GEO tool numbers as a prioritization input, and flags what to watch in the Korean market.
- What Is GEO: The Definition of Generative Engine Optimization and How It Differs From SEOGEO (Generative Engine Optimization) is the strategy of getting your content cited in answers produced by generative engines like ChatGPT and Perplexity. Here is the definition, how it differs from SEO, and how it works.