LinkedIn
Your 37.4% AI visibility score measures what, exactly?
AI visibility scores are Quality Score with worse math. Your brand is at 37.4%, up 2.1 points from last Tuesday, and nobody in the room can tell you what the denominator was. The number arrives on a slide or inside a $149-a-month dashboard, dressed in charcoal grey with a bright green arrow. You are supposed to nod at the decimal place and conclude that whatever your team published last week worked.
Maybe it did. The score cannot tell you.
Look under the hood and the arithmetic changes from tool to tool. As technical teardowns of prompt tracking tools show, Peec divides brand mentions across every prompt run; Profound excludes prompts where no brand was mentioned; Otterly plots a subjective two-dimensional index; and PromptWatch deducts points for competitive context. Put the same domain into three tools on the same morning and you can get three conflicting answers, each presented to a tenth of a percent.
That is not precision. It is a formatting choice.
I watched Quality Score become a client presentation
I spent years managing Google Ads accounts through the spreadsheet era. In 2016, performance marketing had its own magical composite number: Quality Score. Agency decks celebrated a rise in average keyword scores from 6.2 to 7.1. Clients cheered, retainers renewed, and almost nobody asked whether closed-won pipeline had moved. Google’s documentation describes Quality Score as a keyword-level, 1-to-10 diagnostic comparing ad quality with other advertisers. It is not an input into the ad auction or a measure of business profit.
I knew the number could swing while something more important went wrong. A client could change a landing page without telling the account manager. Cost per acquisition could climb 40% while the deck still displayed an 8-out-of-10 score. That score made a convenient answer when the harder work was checking negative lists, conversion tracking, and whether a bid cap was starving high-intent auctions.
An AI visibility score offers the same refuge: a technical-looking proxy that delays the question of whether the brand made any money. The difference is that Quality Score at least referred to a defined keyword diagnostic. Here, the prompt list and even the denominator can change depending on who sold you the dashboard.
A few dozen prompts cannot stand in for a market
Traditional paid-search share of voice had an auditable denominator: Google reported auction impressions for a targeted keyword list and the share your ads captured. Generative engines have no equivalent fixed inventory. As Search Engine Land noted about AI share of voice, a percentage share of an open-ended conversational universe is a different proposition. A buyer might ask about migration timelines from HubSpot, add a European data-residency constraint, then request a shortlist. That is not one static keyword with one static results page.
Yet a tracker may run 25 to 50 fixed prompts once a week, count mentions, and present the result as your market standing. Fifty synthetic queries on Tuesday tell you how the engines answered those fifty sentences on Tuesday. They do not tell you what every buyer sees.
If a competitor appears on 18 prompts this week and 19 next week, a chart can turn that one extra appearance into a 2.0-point surge. Cue two hours in an all-hands meeting about the breakout. The engine may simply have pulled from a different cached index.
The answer changes when you ask again
Generative answers are not stable ranking positions. Even when an identical prompt is run under tightly controlled conditions, the wording can change: an experiment documented by Promptly produced 80 distinct completions from 1,000 runs at temperature 0. In a 90-day benchmark of eight AI search engines, identical buyer prompts kept the same top-three recommended brand set only 46% of the time from run to run.
So a weekly ping is a snapshot of a moving answer, not a permanent ranking. Ask again and your brand may disappear from the summary. A rise from 34% to 41% could reflect a change worth investigating, or it could be ordinary variation. One run cannot distinguish the two. Measuring wind speed once on Monday and calling it the quarter’s climate would look ridiculous. Put the result in a polished dashboard and apparently we need a meeting.
Blending engines makes the number murkier. ChatGPT, Google AI Overviews, Claude, and Gemini do not draw on identical systems. Cross-model brand visibility research found that only 11% of tracked brands appeared across all three major models, while 77% appeared in only one. Notion was cited 17 times more often in Claude than in Gemini. If your brand appears in Claude but not Google AI Overviews, a blended 37% conceals both facts. That is the information you needed.
Someone chose the prompt list
Before trusting a share-of-voice chart, ask where its prompts came from. An agency might brainstorm forty conversational queries; a tool might use your sitemap to generate a list of ‘typical commercial search prompts’. Neither method establishes that a buyer with procurement authority asks those questions.
The list can also flatter the client by design. Mix twenty prompts about your specific features with ten generic queries and, unsurprisingly, the brand shows up. That is confirmation bias with an API bill. In paid search, nobody should buy exact-match keywords for a client’s own name and call the resulting impression share proof of market-wide demand. Dressing self-selected AI prompts in an executive dashboard does not improve the trick.
The prompt list matters because it decides what the score can see. Leave out the question a buyer asks two days before signing, and you can have a beautiful upward line while losing the answer that mattered.
A green arrow is not an action plan
Say your score falls from 38.2% to 34.1%. What do you do before lunch? A discussion on r/SEO_tools_reviews describes composite visibility scores and share-of-voice charts as more useful for client slides than for deciding what to change. I can see why. The drop does not tell you whether a competitor published a useful teardown, an engine refreshed its index, forum sentiment changed, or that week’s run produced a different answer.
You cannot fix an undifferentiated composite. If an agency cannot identify what changed, it cannot tell you what to build, rewrite, or repair. You get ‘build entity authority’ or ‘expand thought leadership’, which is what a recommendation sounds like when nobody wants to name a page. If that pitch sounds familiar, read this open letter to the owner about to buy an AI visibility dashboard before cutting a cheque.
A dashboard can flag a symptom. It cannot fix a blocked crawler, update a page, or adjust a bid. Paying to watch the weather change is still paying.
The smaller test I would actually run
Five buyer prompts, repeated rather than multiplied
I would start with five to ten questions real buyers ask close to a demo or contract, with the commercial intent and constraints left intact. Run the same prompts across the engines that matter to your business, then sample each prompt 10 to 20 times across different days and hours. Do not turn one Tuesday answer into a market-share claim.
We set out a broader version of this approach in the 30-prompt test for AI search visibility. The point is not that thirty is a magic number. Repeated runs show whether a brand appears reliably for a useful question; a single weekly run shows what happened once. I would rather know the answer to five buyer questions repeatedly than admire one composite score built from 200 prompts I cannot defend.
Count mentions, citations, and recommendations separately
A tool can register a ‘win’ whenever an assistant includes your name somewhere in a long answer. That collapses three different events into one numerator:
- Mention: Your brand appears as item eight in a list of twelve vendors, without a link, context, or endorsement.
- Citation: The engine points to your site as a source and provides a clickable link that a buyer can follow to check details.
- Recommendation: The answer weighs the buyer’s constraints and presents your business as the preferred choice.
An unlinked name-drop and a linked recommendation should not count as the same outcome. Separate the three, or the headline number will hide the useful part. Our guide to building an AI visibility tracker covers measuring mention rates alongside citation rates rather than saving full-page screenshots. A recommendation on a core commercial prompt tells me more than a long list of raw mentions on informational ones.
Check what changed before claiming a win
A citation-rate drop in ChatGPT means little without a change log. Did someone update robots.txt and block GPTBot? Did marketing rewrite a comparison page and remove structured pricing data? Did a competitor publish a technical teardown? The log cannot explain every swing, but without it you cannot even begin to separate an operational cause from answer variation.
This is the fix I want: connect monitoring to execution, not another reporting meeting. With groas, specialized AI models work across paid and organic search continuously, including technical adjustments, landing-page changes, and bids. Actions and their reasoning are logged, while a named human strategist sets commercial guardrails. That is a better operating model than paying an agency to narrate a monthly chart: you can inspect what changed and judge the work against attributable revenue.
Three questions before you pay for the decimal
Before buying an AI visibility subscription or approving an agency’s monthly ‘AI Share of Voice’ slide, I would ask:
- What is the denominator, and how often was each prompt sampled? If the vendor cannot explain its formula or ran each prompt only once that week, the decimal place is doing more work than the sample.
- Do you distinguish a mention from a citation and a recommendation? A passing name-drop in ten options does not carry the same commercial weight as a linked answer that recommends you for the buyer’s constraints.
- When the number moves, what executes the fix? A chart can flag something to investigate. A monthly recommendation meeting cannot, by itself, repair a page, resolve a crawl block, or change an auction bid.
Founders sometimes ask me which software will automate AI search visibility reporting without hiring an agency. I think that automates the wrong layer. A faster chart of answers you sampled once does not make those answers more reliable, and it certainly does not make the next change for you. I want repeated checks on buyer prompts, a record of what changed, and execution accountable to pipeline rather than cosmetic percentages.
The decimal places reset when you refresh the page. Closed-won revenue does not.