Why AI Visibility Tools Report Different Scores for the Same Brand: Causes and a Verification Procedure

Measure the same brand in the same week with two tools and the scores will not match. The first reaction is usually "which tool is right," and that question goes nowhere. The two numbers came from different prompt sets, run a different number of times, divided by different denominators. This page splits the reason AI visibility tools disagree into four layers, engine, collection, definition, and aggregation, then walks through the verification steps a dashboard number should clear before it lands in a report.

Four layers behind the gap, defined in 30 seconds

Engine variance is the tendency of an AI engine to return a different answer, and different sources, on each run of the same prompt.

Collection design is the rule set that decides which prompts get sent, how many times, and from which region and login state.

Metric definition is the counting rule that fixes what counts as one appearance and what goes into the denominator.

Aggregation is the weighting and averaging method that folds per-prompt and per-engine results into a single score.

The four choices multiply. Two tools that decide each layer a little differently will report different numbers for the same brand. Visibility as a design variable rather than a fixed constant was already an assumption in the research that first formalized GEO. That paper defined visibility metrics inside generative engine responses, proposed a black-box optimization framework, and reported visibility gains of up to 40%.^[13]

No engine answers the same question twice

The first layer sits outside any tool's control. SE Ranking pushed the same 10,000 keywords through Google AI Mode three times on one day and compared 9,734 responses carrying 122,617 links. Exact URL overlap across the three datasets averaged 9.2%, and 2,006 keywords (21.2%) shared no URL at all across the three runs.^[1]

Every figure comes from (SE Ranking, 2025). The per-pair values scatter between 18.5% and 19% for URLs and between 26.6% and 27% for domains, which is the range the bar chart above reports. Any two answer sets shared 3.3 identical URLs and 3.7 identical domains on average.^[1]

A single response carried 12.6 links on average.^[1] That is roughly a dozen citation slots, redrawn almost from scratch on every run. At the brand level the swing shows up even more plainly. SparkToro compiled 2,961 runs of 12 prompts across ChatGPT, Claude, and Google AI answers by 600 volunteers, and reported that two runs return the same brand list less than once in 100 attempts, with the same list in the same order closer to once in 1,000 (collected November and December 2025).^[3] A separate experiment ran 12 prompts 100 times each on logged-out free ChatGPT, 1,200 runs in total from different IP addresses, and found roughly 44 brands mentioned across the 100 responses per prompt, of which only about 5, near 11% of the total, appeared in more than 80% of those responses (Search Engine Land, 2026).^[4]

The cause sits in inference infrastructure, not service policy. Thinking Machines Lab sampled one prompt 1,000 times at temperature 0 and got 80 distinct completions; the first 102 tokens matched, then the 103rd token split. The main driver was not floating-point nondeterminism by itself but kernel operations whose results change with batch size, and swapping in batch-invariant kernels made all 1,000 runs identical.^[2] API documentation says the same thing. OpenAI states that setting a seed does not guarantee determinism and that backend configuration changes should be monitored through system_fingerprint.^[10]

Platform differences are documented too. Google explains that AI Mode and AI Overviews can run on different models and techniques, so the answers and link sets they surface diverge.^[9] One number labeled "Google AI visibility" cannot be interpreted without knowing which surface it measured. Multi-engine measurement methodology covers the per-engine approach in more detail.

Every tool divides by a different denominator

The second cause is the formula. Put the public documentation side by side and the same metric name turns out to describe different arithmetic.

Price is left out of the comparison axis. What splits the numbers between tools is the formula and the collection design, not the plan tier, and those are also what decide whether a result can be reproduced in practice. Public pricing moves, so check each vendor's official page during evaluation. Price bands are collected in AI visibility monitoring tool comparison.

The size of the denominator effect is easy to check with arithmetic. Say 60 of 100 responses name at least one brand, and 18 of those name yours. Divide by all responses and the score reads 18%. Divide by responses that name at least one brand and it reads 30%. Same data, one definitional change, 12pp apart. This example is a hypothetical constructed to show the denominator effect, not output from any specific tool.

One more axis matters. A brand mention and a domain citation are different events. Your name can appear in an answer while your domain goes uncited, and your page can be used as a source while the brand name never shows up in the sentence. That is why Ahrefs documentation separates mentions and citations into distinct metrics and states that citations count only pages shown as inline sources in the response, excluding pages that were crawled but not cited.^[7] Align the metric names without aligning the event definitions and the two tools will keep producing different answers. Ahrefs Brand Radar vs Semrush AI Toolkit sets the two products' metric designs side by side. AI search share of voice picks up how share metrics are structured.

With a small sample, most of the gap is noise

The third cause is the sample. A visibility score is a proportion estimate, so a small sample means a wide confidence interval. Compute the 95% interval for a binomial proportion with the normal approximation and the width follows from the observed rate and the sample size. The figures in the table below are not any tool's reported output, they are computed directly from that formula. The NIST/SEMATECH handbook notes that this approximation is the most commonly used one but calls it inferior, because its lower limit can drop to an impossible value, and recommends the Wilson method instead, so treat the table as an order-of-magnitude guide.^[12]

Response sample size here is prompts times runs times engines. Fifty prompts run once on one engine gives a sample of 50, and at that size the move from 28% last week to 33% this week supports no claim about performance. GEO KPI measurement goes further into repeat measurement and prompt panel design.

Grading measurement quality has already drawn a standardization attempt. In August 2026, IAB's "Measuring Visibility in the AI Era" split AI visibility data into directional and decision-grade, and set out the principle that measurement vendors disclose their query selection method, collection cadence, and aggregation methodology.^[11] The full framework is explained in IAB Four Ps measurement standard.

Verification, in six steps

None of this argues for throwing the numbers out. It argues for drawing a line around how far they can be trusted. The sequence starts by pinning down the spec.

Steps 1 and 2 are the ones teams skip most often. Without a spec, nobody can later reconstruct the conditions a number came from. Without definitions, tool-to-tool gaps keep getting mistaken for arguments about performance.

Step 4 needs controlled conditions. Make logged out, fixed region, fresh session, and no personalization history the defaults, and record the conditions used. SparkToro disclosed that its study did not align temperature, country, device, or chat history and used whatever default experience each participant already had, and that disclosure is exactly the kind of information that changes how a result reads.^[3]

Step 5 runs faster in order. Match engine and model first, then prompt set and run count, then the denominator. Whatever survives those three often traces back to aggregation weights. Merging multiple platforms with an impression-weighted average, for example, lets high-volume prompts dominate the score.^[7]

Verifying visibility from log data confuses the layers. Google states that performance for sites surfaced in AI features is folded into the Web type within total Search Console search traffic and is not broken out separately.^[9] Traffic and conversion data describe what happens after a citation, so manage them as a different metric from in-answer mention rate.

What else splits in Korean-language measurement

Korean brands carry one more variable: query language and the tracked engine set. The same brand draws a different answer mix from English and Korean queries, and when tools track different engine lists, the denominator itself changes. Global tools work well for tracking the major overseas engines, while coverage of domestic Korean engines varies by product, so confirm it against official material before buying.

Domestic options are growing. BOIDA, run by Designovel, states that its BVI metric measures brand exposure in major generative AI answers and carries that through to remediation as part of its product scope^[14]. Press coverage describes it as analyzing exposure across major generative AI from a URL or product name alone (AVING, 2026)^[15]. The named engine list and the way Korean-language queries are handled are best confirmed against official material. Other domestic vendors include Next-T, Across, LeadGenLab, and Ascent AI; what each one publishes, and where its free tier ends, is laid out spec by spec in the comparison of free GEO audit tools in Korea. Ranking or first-place phrasing that Korean vendors apply to themselves is subject to fact-checking, so read it as claimed positioning and compare instead on documented engine lists, query selection methods, and disclosure of collection cadence.

Takeaways

The gap between tools is not a bug. Engine answers change from run to run, tools send different prompts different numbers of times, and metrics sharing a name divide by different denominators, so the gap is a structural outcome. The 9.2% exact URL overlap across three same-day Google AI Mode runs, on its own, sets the resolution of any single measurement.^[1]

Three conclusions carry into practice. First, do not compare absolute values across tools. Read only the trend and the relative competitor position inside one tool under one spec. Second, attach the spec and the confidence interval to any number that goes into a report; one line listing prompt count, run count, engines, period, and denominator definition makes next quarter's reading far easier. Third, put methodology disclosure into your tool selection criteria. A product that publishes its formula and collection design is a product you can verify. What is GEO covers the definition and scope, and AI visibility monitoring tool comparison continues on tool choice.