LinkedIn
Ask ChatGPT for the best CRM for a small law firm on Tuesday, then ask again Thursday with the exact same words. Two names vanish; a new one appears. I have watched clients forward both screenshots in the same email and ask which answer is true.
Neither screenshot is a fixed record of where the brand stands. ChatGPT does not look up a stored answer; it samples next tokens probabilistically. Each prompt draws a fresh answer. A single check of “does it mention us?” tells you as much as one day of data tells you about a headline test: almost nothing. To get a useful visibility number, you have to understand what varies, hold steady what you can, and count enough draws to see past the noise.
The answer is written fresh, not retrieved from a file
Start with the one fact everything else follows from: there is no answer file sitting on a server waiting for your question. The model writes an answer one token at a time. It ranks likely next tokens, then draws from that ranking rather than always choosing the same one. Change an early choice and the rest of the answer has a different path to follow.
Temperature affects that draw, but it is not the only source of variation. Floating-point rounding, GPU parallelism and how a request gets batched with others can matter too. That is why even temperature 0 is not guaranteed to produce the same answer.
The providers say as much. Anthropic documents that results are not fully deterministic even at temperature 0.0, OpenAI calls its seed parameter best-effort only, and Thinking Machines Lab traced leftover variation in September 2025 to server load and batch arithmetic changing the next-word choice. I used to tell clients to set temperature to zero for a stable check. I was wrong. Zero narrows the spread; it does not freeze the draw.
For our law-firm CRM prompt, that means the model can produce a different list before anything about your brand or its competitors has changed. One answer is one draw, not a verdict.
Live fetching changes the material the model sees
The sampled writing may sit on top of live web fetching. When a model grounds an answer, it pulls pages available at that moment, and it can fit only so much retrieved material into the answer process. One teardown of 7,060 queries put the grounding budget near 2,000 words, with chunks averaging 15.5 words. A small ranking move or a page falling outside that window can change which facts the model sees.
Run the same CRM prompt twice and the model may read two slightly different piles of sources, then sample two summaries from them. The answer can move because the input moved. Do not treat every missing mention as evidence that your page got worse.
One question can become several searches
Google confirms that AI Overviews and AI Mode may use query fan-out, issuing related searches across subtopics and data sources. A question about a small-firm CRM can branch into legal practice management, document automation and pricing pages. Each branch may return a different set of names. Combine shifting search results with a sampled answer and the brand lineup shuffles again.
The ad-auction analogy gets you part of the way: the competitors in view can change from one run to the next. It breaks at the query. Here, the system may change the searches behind your question, too. If you want to measure your presence, record the question you asked, but do not assume the system took the same route every time.
Your session changes the question too
Memory, custom instructions, location and chat history can all affect the answer. Lily Ray showed this in November 2025: “Where to shop for trusted brands in electronics” became a search for “electronics retail trusted brands NYC store” when run from a New York location, with different brands and citations attached.
Test logged out in one city and logged in from another, and you are not testing the same conditions twice. Your spreadsheet may show identical prompt text; the model is receiving more than that text. Fix the session and location before you compare runs.
A mention is an event; visibility is a rate
Once answers can vary, a single yes-or-no check stops being a visibility score. What you can estimate is a mention rate under stated conditions: how often your brand appears when you repeat a prompt on a particular engine, in a particular mode and setup. Change those conditions and you may get a different rate. There is no universal visibility number that floats above model, retrieval, moment and user.
I have seen teams rewrite a page because Monday said mentioned and Tuesday said missing. That is not evidence of a drop. It is sampling noise wearing a dashboard costume.
The spread is bigger than most people expect. SparkToro ran 2,961 queries with 600 volunteers and found the same brand list repeated less than 1% of the time. Its estimate was around 1,000 runs before two identical ordered lists appeared. A Washington State test from March 2026 found consistent answers about 73% of the time when running the same prompt ten times.
Mike Sonders pushed on brand presence specifically. Twelve buyer-intent prompts, each run 100 times in logged-out ChatGPT from different IPs, surfaced about 44 different brands per prompt. Roughly ten appeared in each answer; only about five brands appeared in 80% of responses or more. Of the brands mentioned at all, 72% appeared in fewer than one in five responses. A brand can be visible without being reliably present in any single answer. That is exactly why one screenshot is a poor report.
Read the band before the week-over-week line
A Brand24 roundup puts the ordered-list problem plainly: the same brands in the same order appear in about one in 1,000 runs. The percentage across repeated runs is worth tracking. The one-time list is not.
Say our law-firm CRM appears in three of five checks one week and two of five the next. That is 60% followed by 40%, but it is not a reason to announce a decline. With only five runs each, the estimate wobbles too much to support that call. The same goes for a jump from two of five to four of five. The numbers changed; you have not yet shown that the underlying rate did.
Read the uncertainty band first, the chart line second. If a report gives you a precise-looking weekly score without showing how many runs produced it, ask for the denominator before you change a page.
Treat each run like an impression, not a result
I ran headline tests in Google Ads for years, and the measurement problem is familiar. Nobody I trusted would call a winner after 40 impressions per variant. We waited for a few hundred impressions per headline, then asked whether the gap was bigger than the wobble.
For AI visibility, the prompt plays the role of the ad group, a run plays the role of an impression, and a mention is the event you count. The analogy is about repeated observations before making a call. It is not a claim that an AI mention has the business value of an ad conversion, or that either system holds all other conditions perfectly still.
Say you spend $20k a month and judge a landing page on 11 clicks. You would laugh. Judging your AI presence on three ChatGPT checks deserves the same reaction. Count mentions per hundred runs, not mentions per screenshot.
Test protocol: turn the CRM prompt into a measurement
Question: For “best CRM for a small law firm,” how often does ChatGPT mention our brand? If that rate moves, will the test let us notice?
My expectation before running it: the first ten answers will look chaotic; the estimate will become more useful as the runs accumulate; and a 15-point swing based on just 30 runs will not be enough for me to call a change. The mechanism is the sampling and shifting context above. More repeats reduce the noise in the estimate. Write down your expectation now, before the first result gives you a story you want to believe.
Setup and controls:
- Pick five prompts buyers actually type: a best-of list, a comparison, a problem question, a pricing question and an alternative-to-competitor question. Keep the law-firm CRM prompt as the running example.
- Run each prompt 30 times per engine to start, across ChatGPT, Gemini and AI Overviews. Use a fresh logged-out session, fix the location, clear chat history and note the model version.
- Record every answer as mentioned or not mentioned. Do not add partial credit or sentiment adjustments yet. Keep the prompt text and setup the same when you repeat the measurement.
- Continue to 60–100 runs per prompt per platform before leaning on the rate. That is the range other testers keep landing on; it is not a guarantee that every comparison will be settled.
The controls will not stop the model or its sources from changing. They stop you from adding avoidable changes and then calling the result an AI visibility trend. Log the conditions beside the answers. Otherwise you will not know what you compared.
Calculate the rate, then show its wobble
Count mentions and divide by runs. If the CRM appears in 18 of 30 answers, its observed mention rate is 60%. Now put an error band around that estimate before you show it to anyone. At 30 runs, a rough 95% band around 60% extends about 18 points in either direction: roughly 42% to 78%. At 100 runs, the band around a similar rate tightens to about ten points either way.
Those bands describe uncertainty in an estimated rate, not the boundaries of what the next answer can say. They also assume you kept the test conditions comparable. This is why five checks across each of a handful of prompts give you observations, not a dependable score. Same rule I used for ad tests: no band, no call.
If the law-firm CRM shows 40% one week and 55% the next on 30 runs each, I would not call that growth. The estimates are too imprecise. If it moves from 35% to 65% on 100 runs each and stays there for two cycles, I would investigate what changed. The repeated gap is more useful evidence than either week alone. Never forward a visibility screenshot without its denominator.
Decide what either result changes
A test that cannot change what you do next is theater. If the mention rate sits at 30% on 100 runs and a month later sits at 32% on 100 runs under the same conditions, change nothing on that evidence. If it moves from 30% to 55% on the same prompts and engines, then holds for two cycles, inspect the sources and the pages that may have affected them.
That is where I would split pages into a holdout: fix half, leave half alone, then re-run. Readers who want the split written out can follow the AI visibility holdout test. The point is to find out whether a change beats the band, not to credit the last page someone edited.
Keep the instrument consistent, too. An OpenAI API check with your own system prompt and tool settings is not consumer ChatGPT with its routing, search logic and shopping modules. There is no developer API for AI Overviews or AI Mode. I learned this the annoying way, comparing a clean-looking API log with a browser check that named three competitors the API never showed. Measure where your buyers ask. For this setup, that means browser checks with location fixed, history cleared and the version noted each time.
Automate the repeats, not the false certainty
You do not need to hand-run 3,600 checks. Sixty runs across 20 prompts and three platforms already gets you there. Repetition is what software is for. Tracking tools re-run prompts over time across surfaces including ChatGPT, Gemini, Perplexity and AI Overviews, often adding share of voice and sentiment. The cited roundup lists prices from about $20 a month for Rankscale and $29 for Otterly to $99 for Semrush per domain, $199 for Ahrefs, $300 for Scrunch and $800 for Evertune.
The price list is less important than three questions: Does the tool repeat prompts enough? Does it keep conditions comparable? Does it show the denominator and uncertainty instead of selling you a solitary “78” score?
groas tracks prompts with answers, citations and share of voice checked weekly, then acts on the gaps it finds rather than stopping at the dashboard. If you want to see the measurement before automating it, run this mention audit and log the repeats. Then look at a tracker built around mention rates, not screenshots. Do not pay anyone to email you noise faster.
Now take a new case. Say you sell scheduling software to dentists, and the buyer prompt is “best scheduling tool for a two-chair practice.” One check includes you. The next leaves you out. You know from the model we built that neither answer, alone, tells you your rate: the token draw can change, the fetched pages can change, the searches behind the prompt can branch differently, and the person running the check may be in another city with another session.
So fix the conditions and run the prompt 100 times per engine. Suppose your brand appears in 25% of answers, with a band of roughly ten points either way. You can stop arguing about Tuesday versus Thursday. You can also stop pretending that 25% identifies the cause. Look at the sources behind the answers, compare the brands that appear consistently, and test a change against a holdout. That gives you a work order instead of a worry.
Stop asking whether the answer mentions you. Ask how often it mentions you, under which conditions, with what band, and whether that rate holds when you repeat the test. Anything less is one day of ad data dressed up as a report. I stopped buying those reports years ago.