Measuring AI visibility: a methodology guide
A trustworthy AI visibility number needs five things: a clear metric definition, a representative prompt set per language, repeated sampling of each prompt on each engine, a confidence interval next to every rate, and classifiers (mention, tone, claims) whose accuracy has been measured on labelled data. Without them, a change in the number may be noise.
Which metrics should you define first?
Write the definitions down before collecting any data, so that the numbers mean the same thing every week:
- AI visibility rate: answers mentioning the subject ÷ answers sampled.
- Recommendation rate: answers explicitly recommending the subject ÷ answers mentioning it.
- Share of voice in AI answers: mentions of the subject ÷ mentions of the subject and its competitors.
- Tone in AI: net tone towards the subject in the answers that mention it, for example from −100 to +100.
- Claim accuracy: the share of factual claims about the subject that match its verified facts.
Decide also what counts as a mention: exact names only, or partial names and descriptions such as "the Italian capsule brand"? Both are defensible; switching between them silently is not.
How should the prompt set be designed?
The prompt set is your sample frame, and it drives every result. A good set:
- covers the categories of questions people ask: discovery, comparison, evaluation, price, support, current events;
- is phrased like real questions, not like search keywords;
- is translated and localised for each language and country tracked, and each language version is treated as a separate prompt;
- is versioned: when you add or change prompts, results before and after are not directly comparable, so record when it happened.
Report results per prompt category as well as overall. A brand can be highly visible on "X vs Y" questions and absent from "best X" questions, and an overall average hides that.
How many samples do you need?
AI answers vary between runs, so each prompt must be run several times on each engine. The question is how many. For a proportion, the width of the 95% confidence interval depends mainly on the number of answers:
| Answers sampled | Observed visibility | 95% Wilson interval (approx.) |
|---|---|---|
| 10 | 60% (6 of 10) | 31% – 83% |
| 40 | 60% (24 of 40) | 45% – 74% |
| 100 | 60% (60 of 100) | 50% – 69% |
| 400 | 60% (240 of 400) | 55% – 65% |
Two practical consequences. First, a single prompt run a handful of times cannot support a precise number; aggregate across prompts or across weeks when you need precision. Second, a change between two periods is meaningful only if it is larger than the intervals suggest; a move from 60% to 55% on 40 answers is well within noise.
The Wilson interval is a good default for rates because it behaves well with small samples and with rates near 0% or 100%, where the simpler normal approximation fails. For averages such as tone, use an interval on the mean.
Why do sampling conditions matter?
Record and hold constant the conditions of each sample: engine, model version when available, whether web search or grounding was on, language, country, and date. Consumer apps add personalisation, memory and location, which makes them hard to reproduce; official APIs give more controlled conditions but can differ from what a given person sees in an app. Either choice is defensible if it is stated. Mixing them is not.
How do you know the classifiers are right?
Every metric beyond raw mention counts depends on a classifier: is this answer about our subject (and not a namesake)? What tone does it take towards it? Which claims does it make? Whether that classifier is a person, a rule or a model, its accuracy must be measured:
- build a labelled benchmark of real answers and pages, in each language you report on;
- measure precision and recall for mention and relevance detection, and macro F1 for tone;
- never evaluate a model on the same examples it was trained on;
- set targets before looking at results, and publish the measured values next to them;
- version every score with the model that produced it, so a better model can re-score history without silently changing past reports.
Can you attribute answers to sources?
Cited pages are certain. The influence of uncited pages is an estimate and should be labelled as one, with its components visible. When testing whether sources lead answers over time, beware of multiple comparisons: test enough pairs of series and some will correlate by chance. Use a significance test suited to time series (for example a permutation test), correct across all the pairs tested (for example with the Benjamini–Hochberg procedure), and require enough data points before claiming a lead. Even then, a lead is a correlation, not proof of cause.
What are the common pitfalls?
- Reporting a single answer or screenshot as "what ChatGPT says".
- Averaging across languages that behave differently.
- Changing the prompt set and comparing before and after as if nothing changed.
- Counting a namesake as a mention.
- Reporting tone without saying whether it is tone towards the subject or tone of the whole answer.
- Omitting intervals, so every wobble looks like a trend.
Questions to ask any AI visibility tool
- How many times is each prompt run on each engine, and how often?
- Are answers collected through official APIs or from consumer interfaces, and with web search on?
- Are confidence intervals shown?
- How are mention, tone and claim classifiers validated, in which languages, and against which targets?
- Are scores versioned, and can history be re-scored?
- Are the full answers kept, so any number can be audited?
Tonecast publishes its answers to these questions on its methodology page and on a methodology card for every campaign: weekly sampling of ChatGPT, Claude, Gemini and Perplexity through official APIs, 95% intervals on every rate, published quality targets, and versioned, re-scorable models.