Why AI Recommends Different Brands Every Time You Ask the Same Question: 5 Causes of Response Variability and How to Measure Around Them

"Yesterday our brand showed up when we asked the AI. Today, same question, it's gone." Almost every team that starts auditing AI visibility sends some version of that message. The person asking assumes something broke and goes looking for the fault, but usually nothing broke. Each AI answer is generated fresh, and several stages of that generation produce different values on every run. This article separates the causes of shifting brand recommendations into five layers, uses public experiments to show how much each layer moves, and then sets out a five-step procedure for measuring visibility on top of answers that never sit still.

30-second definitions: five terms behind response variability

Response variability is the tendency for an AI's answer content, mentioned brands and ordering to differ from run to run when the same question is asked of the same AI.

Sampling is the generation method in which a model extends a sentence by randomly picking one token from a probability distribution over candidate next tokens.

Temperature is the setting that controls how random sampling is; the closer it is to 0, the more the model picks the highest-probability token.

Inference nondeterminism is when identical input produces different output even at temperature 0, because compute conditions on the server differ.

Query fan-out is how AI search splits a user's question into multiple subqueries, runs them in parallel, and assembles the results into one answer.

These five terms follow the order in which an answer gets built. Prompt wording and personalization context change the input, fan-out search changes the reference documents, and sampling plus inference nondeterminism change the final sentences. Small differences at each layer multiply, and the brand list at the end comes out different.

Evidence: how often does the same question return the same brand list?

The largest public experiment is a study SparkToro published in January 2026. Six hundred volunteers ran 12 prompts on ChatGPT, Claude and Google AI (AI Overviews, or AI Mode when no Overview appeared) a combined 2,961 times, with data collected in November and December 2025. The headline finding: ask ChatGPT or Google AI 100 times, and the chance that two responses give the same brand list is under 1%; the same list in the same order shows up roughly once in 1,000 runs.^[1] The study noted that Claude was somewhat more likely to repeat the same list.^[1]

Even with the lists changing every time, brands that appeared often kept appearing often. Across 142 human-written headphone recommendation prompts and 994 responses, top brands such as Bose, Sony, Sennheiser and Apple appeared in 55 to 77% of responses.^[1] Rankings were another story. One hospital appeared in 69 of 71 ChatGPT responses (97%) but was listed first in only 25.^[1] That result is where measurement design starts. Lists and ordering move around, yet the rate of appearance across many runs still carries a readable signal.

Causes 1 to 5: at which layer do brand recommendations change?

The five causes behave differently. Users can control some of them; only the engine operator can change others. A sound measurement design pins down the layers you can control and averages out the ones you can't through repeat runs.

Cause 1. People almost never phrase a question the same way

The single prompt a measurement tool sends is not what real users type. In the SparkToro experiment, the 142 human-written headphone recommendation prompts had an average semantic similarity of 0.081.^[1] Two questions with the same goal but different budgets, use cases or comparison targets lead the model to consider different sets of candidates. The study expected real-world variability to exceed what the controlled experiment showed.^[1]

Cause 2. Personalization context changes the input

In April 2025, ChatGPT announced a memory upgrade that references a user's entire chat history, adjusting answers to their preferences, writing style and recurring questions. Users can turn memory off or use Temporary Chat, which neither references existing memories nor saves new ones.^[6] A marketer who repeatedly searches for their own brand from a personal account may be feeding that chat history into later answers. What you see on your own account does not represent what anyone else sees.

Cause 3. Fan-out search changes the reference documents

For search-based answers, this layer moves the most. Google said in May 2025 that AI Mode uses a query fan-out technique, splitting a question into subtopics and issuing multiple queries at once.^[4] Ahrefs wrote that these fan-out queries are simulated, change on every run, and that Google does not disclose which fan-out queries it used for any given answer. The same analysis found that 37.9% of URLs cited in AI Overviews fell within the top 10 blocks of the original search results, down from about 76% in a July 2025 study. Ahrefs also noted that it had improved its parsing methodology since the first study.^[5] Change the subqueries and you change which documents get retrieved, and with them which brands appear.

Cause 4. Sampling produces different text by design

An LLM computes a probability distribution over the next token and then draws one from it. Google's Vertex AI documentation explains that temperature controls the degree of randomness in token selection, with higher values producing more varied output.^[3] If the first brand in a list changes, every sentence after it follows that choice and shifts too. Consumer chatbot apps don't let users set this value.

Cause 5. Even at temperature 0, output is not fully locked

The same Vertex AI page says that temperature 0 always selects the highest-probability token, yet responses are only mostly deterministic and a small amount of variation can remain.^[3] In September 2025, Thinking Machines ran one prompt 1,000 times at temperature 0 on the Qwen3-235B model and got 80 distinct completions. All of them matched for the first 102 tokens and diverged at token 103. The researchers identified batch size, which changes with server load, as the main cause; switching to batch-invariant kernels made all 1,000 completions identical.^[2] This experiment used an open model. You can't map these numbers onto the internal settings of commercial services such as ChatGPT or Gemini, but it does confirm that server-side conditions can change output even when the user fixes the value.

Telling real change from noise

The question a practitioner actually needs answered is whether a different result is chance or a sign that something really changed. The table below sets out criteria for that call, separating a single observation from repeat measurement.

Evertune published how far the margin of error shrinks as you repeat the same prompt. In an analysis that repeatedly ran an SUV recommendation prompt on ChatGPT, the margin of error was 27pp at 5 repeats, 12pp at 12 and about 6pp at 100, with diminishing returns past 100.^[7]

A 27pp margin of error at 5 repeats means a brand observed at a 10% mention rate could truly sit anywhere from 0% to 37%.^[7] Any judgment made from asking by hand a few times falls somewhere inside that range.

Response: five steps for measuring visibility on answers that keep moving

Step 1, design a prompt set. Real users' questions barely overlap,^[1] so don't rely on one representative prompt written as a single sentence. Group prompts that express the same intent under different conditions, such as "recommend commuter sneakers for men in their 30s," "comfortable sneaker brands for commuting," and "compare commuter sneakers around 100,000 won." Grouping results by intent filters out differences that come from one particular phrasing.

Step 2, control the measurement environment. Turn off memory or use Temporary Chat so a personal account's memory and chat history don't leak into results.^[6] Record location, language and login state as well, so the next round can reproduce the same conditions.

Step 3, run repeatedly. Diagnosing a single prompt requires repeats. SparkToro recommended running each prompt 60 to 100 times and averaging.^[1] For overall visibility, increasing the number of distinct prompts tracks real usage better than repeating one prompt many times. How to split effort between repeats and prompt count continues in Why AI visibility tools report different numbers and GEO performance measurement and prompt monitoring.

Step 4, aggregate as rates. When the same order shows up about once in 1,000 runs,^[1] a single ranking cannot serve as a reporting metric. Tally brand appearance rate and source citation frequency separately for each engine, and calculate your share relative to competitors using the AI search share of voice (SOV) method. Engines search differently, so look at per-engine values first, following multi-engine measurement, before combining them.

Step 5, compare periods. Re-measure with the same prompt set, the same environment and the same number of repeats as last time. If the difference between periods sits within the margin of error, don't report it as a change.

If running this by hand is impractical, use an AI visibility monitoring tool that automates repeat runs and aggregation. Overseas options include Profound, Peec AI and Otterly.ai; in Korea, options include BOIDA, which highlights support for Korean-language queries and Korean engines. When choosing a tool, check how many times it runs each prompt, whether it reports results as rates with margins of error, and whether it discloses its measurement environment. SparkToro likewise advised asking for statistical methods you can review before spending money on a tool.^[1]

Aim to raise appearance odds, not to reduce variability

Brands cannot change sampling or inference nondeterminism. What they can change is the probability of landing on the candidate list. Fan-out search splits a question into subqueries to gather documents,^[4] so having a page that answers each use-case, price-tier and comparison question makes it more likely to be retrieved again and again across runs. Top brands consistently appeared in 55 to 77% of responses,^[1] which fits the same reading: those brands are repeatedly named across many sources as representatives of their category. For how AI picks its sources, see How AI chooses which sources to cite and Citation source patterns by AI engine. If only competitors get recommended, the diagnosis is in Why competitors appear in AI search and your brand doesn't. For the broader concept, start with What is GEO.

Key takeaways

• AI brand recommendations change on every run across five layers: prompt wording, personalization, fan-out search, sampling and inference nondeterminism.
• The same list appeared twice less than 1% of the time, and the same order about once in 1,000 runs (SparkToro, 2026).
• Temperature 0 does not guarantee fully fixed output (Google Cloud, 2026; Thinking Machines, 2025).
• Instead of a single ranking, compare mention rates built from many prompts and repeat runs, period over period, in the same environment.
• Content cannot change the variability itself; it can change the odds that your brand becomes a candidate for each subquestion.