The AI visibility measurement playbook

In the second week of August, Reddit's share of ChatGPT citations fell from 3.83% to 0.52%. Every AI visibility dashboard that counted Reddit-sourced citations as a headline metric reported a catastrophe. Almost none of the brands using those dashboards experienced one, because the engine kept recommending the same brands while swapping which sources it linked. That gap, between a metric collapsing and a business being unaffected, is the entire subject of this playbook.

Why AI visibility measurement keeps breaking

Three structural problems sit underneath almost every broken AI visibility report, and they compound.

The first is that the available metrics are the ones that are easy to collect rather than the ones that matter. Citation counts are easy: an engine either linked a URL or it did not, and that is countable at scale. Whether the engine recommended your product as an answer is harder, because it requires reading the response rather than parsing its links. So the industry standardised on the countable thing, and the countable thing turned out to be the volatile one.

The second is that the underlying platforms change without notice and without changelogs. The August citation shift happened over roughly six days. Google added a new model to AI Mode in mid-August, one day after its general release. Nothing in this environment is stable enough that a metric defined once in January is still measuring the same construct in September. Measurement systems built on the assumption of stability do not degrade gracefully, they invert.

The third is that the analytics plumbing is genuinely wrong in ways most teams have not checked. A nine-month study of 51,200 tracked AI Overview events, published by Alex Galinos at Search Engine Land on August 18, found 22.4% of AI Overview traffic misattributed to the Direct channel rather than Organic Search, varying between 16.8% and 29.3% month to month. If roughly a fifth of the traffic is landing in the wrong bucket, and the size of the error moves every month, then the trend line everyone is reading is partly an artifact of the measurement. The full methodology is published at Search Engine Land, and it is worth noting the sample is a single brand in one industry, so the direction of the error is the transferable finding rather than its exact size. We covered the platform-side version of this in the GA4 AI assistant channel and its limits.

Those four numbers frame the whole problem. A citation metric that fell 86% without a corresponding business impact. An attribution error large enough to swamp most reported changes. And a headline traffic contribution that ranged from 2% to 17% across nine months at a single brand, which means anyone quoting a single figure for how much traffic AI Overviews send is quoting a month, not a rate. The plays below are ordered by how much reporting damage each one prevents.

Play 1: Separate recommendation share from citation share

This is the play that would have made August a non-event, and it is the one most programs do not run because citation counting is what the tools sell.

Citation share measures which URLs an engine linked. Recommendation share measures whether the engine named your brand as an answer. They are different constructs and they move independently, which August demonstrated at scale: Petra Labs reported that overall brand visibility inside ChatGPT responses remained largely intact through a period when citation composition changed drastically across Reddit, YouTube, TikTok, LinkedIn and Facebook. The links churned. The recommendations did not.

The business consequence follows the recommendation, not the citation. A buyer asking an engine which tools they should evaluate acts on the names in the answer. Whether the engine linked your documentation, a review site, or a trade publication underneath that answer affects your attribution reporting far more than it affects the buyer's shortlist. Reporting citation share as the headline number inverts that priority.

The objection is that recommendation share requires reading responses rather than parsing links, and reading does not scale. That is true and it is also the point. Fifty questions across three engines, run monthly, is 150 responses to read. That is a couple of hours of work by someone who understands the category, and it produces a number that survived a platform event that destroyed the automated alternative. Freezing the question wording matters more than the sample size: a question set that drifts is not a time series.

Play 2: Instrument AI Overview traffic before you report on it

This is the plumbing play, and it is where most reporting is quietly wrong rather than volatile.

When a user clicks a cited snippet inside an AI Overview, Google appends a text fragment to the destination URL. That fragment is capturable in analytics as a custom dimension, which is what the nine-month study used to isolate 51,200 AI Overview events without waiting for native reporting. That single implementation decision is what made the rest of the analysis possible, and it is available to anyone with access to their own analytics configuration.

What it exposes is uncomfortable. Beyond the 22.4% average misattribution to Direct, the study found citation traffic extraordinarily concentrated: across 1,661 cited snippets, the average snippet drove 31 events while the single top snippet drove 2,276. That is a distribution where reporting an average is close to meaningless, and where losing one snippet can look like losing a channel.

One caveat worth carrying: this technique measures AI Overviews specifically, because it depends on a fragment Google appends. It does not capture ChatGPT, Claude or Perplexity referrals, which arrive with their own referrer patterns and need separate handling. Anyone presenting fragment-based numbers as total AI traffic is overstating what the method covers, and that overstatement will be found eventually by somebody in the room who knows.

Play 3: Report per engine, never blended

Blended AI visibility numbers are the most common reporting error in the category and among the easiest to fix.

The engines behave nothing alike. Muck Rack's analysis of more than 25 million cited links found ChatGPT citing sources on 96% of responses at an average of 5 citations each, Gemini on 82% at 8 citations, and Claude on 55% at 13 citations. A citation in ChatGPT is one of five slots on a near-certain citation event. A citation in Claude is one of thirteen slots on a coin flip. Averaging those into a single number produces a figure that describes no engine and responds to no action.

The top cited domain differs too: Wikipedia for ChatGPT, Reddit for Gemini, PubMed Central for Claude. Three different theories of what counts as authoritative. The work that wins a slot in one is frequently not the work that wins a slot in another, which is the practical reason blending is harmful rather than merely imprecise. It hides the fact that you are winning on one engine and losing on another, and it makes the resulting number unresponsive to any decision you could make. We walked through the allocation consequences in link building for AI citations, and the divergence is the same story from the measurement side.

Dropping engines from the report is the move people resist and it is usually correct. If your buyers do not use Claude, tracking Claude generates work, noise and occasional panic without informing a single decision. Coverage is not the goal. Decision relevance is. The same logic applies to the forced-search behaviours that distort tool-reported numbers, which we unpacked in why AI visibility tools that force a search skew the data.

Play 4: Baseline volatility before you commit to a target

This play prevents the most expensive category of reporting mistake, which is committing to a number that was a peak.

In the nine-month dataset, AI Overviews drove 7.53% of total organic sessions across the full period. Inside that period the metric peaked at 16% to 17% in February and March, then fell to between 2% and 4%. Same site, same measurement, same nine months. A team that baselined in March and set a target from it committed to holding a number that was roughly four times the eventual trough, through no fault of their own and with no available action to defend it.

This is why the honest form of an AI visibility number is a range with an observation window attached. Not 7.53%, but between 2% and 17% over nine months, currently around 3%. That is a less satisfying sentence and a far more defensible one, and it changes the conversation with whoever is receiving the report from a performance review into a planning discussion. It is also the same discipline behind our client trust report methodology: state the uncertainty in the artifact rather than in a caveat someone will skip.

That last move is the one that protects the relationship rather than the report. Platform changes will move your numbers, repeatedly, in both directions. The difference between a bad quarter and a lost account is usually whether the possibility was discussed before it happened or explained afterwards. Agreeing the threshold in advance costs one paragraph in a scope document and it converts a future crisis into a scheduled conversation. The broader difficulty of proving any of this spend works is real, and we took it seriously in nobody can prove AI visibility spend works.

Play 5: Classify citations by source type, not by domain

The final play converts citation data from a scoreboard into a diagnostic, which is the only role it should have been playing.

Domain-level citation reporting tells you that a given URL was linked. Source-type reporting tells you what kind of authority the engine reached for, and that is the signal that predicts your exposure to the next platform change. The August shift is the clearest possible illustration. A brand tracking domains saw an unexplained collapse. A brand tracking source types saw community sources falling and first-party documentation rising from roughly 2% to 32% of cited sources, which is a legible change with an obvious response.

Six categories are enough: journalism and earned media, first-party documentation, community and discussion, review and comparison platforms, marketplaces and directories, and encyclopedic sources. Classify every citation your question set returns into one of those, and track the mix over time. The mix is the finding. A brand whose citations are 80% community sourced is one platform decision away from a bad quarter, and that is knowable in advance rather than in retrospect.

Putting the AI visibility measurement stack together

Run in order, the five plays produce a reporting stack with a specific property: nothing in it inverts when a platform changes its mind. Recommendation share is the headline and it proved durable through the largest citation shift of the year. AI Overview traffic is instrumented from first-party data with a stated correction rather than an inherited assumption. Every metric is per engine. Every target is a range with a window. And citation data has been demoted from a scoreboard to a diagnostic that tells you where your exposure sits.

What it does not give you is a single impressive number, and that is worth being honest about internally before you present it. The blended, tool-generated AI visibility score is easier to put on a slide and easier for an executive to remember. It is also the number that reported a catastrophe in August that did not happen, and a stakeholder who has been trained on it will need a real explanation of why the new report looks less confident. Do that conversation once, early, rather than during the next platform event.

There is a wider version of this problem that is worth keeping in view. Published estimates of AI search market share disagree with each other by wide margins, which we went through in why the market share numbers disagree, and independent measurements of which engines cite which sources overlap far less than anyone expects, as the citation overlap study showed. Measurement in this category is genuinely hard, and a report that acknowledges that is more credible than one that does not, not less.

Start with Play 1 this week. Write the question list, run it once across the engines your buyers actually use, and record whether your brand was named separately from which URLs came back. That single artifact, a frozen question set with two columns, is the foundation the other four plays build on, and it is the one thing in this playbook that cannot be bought from a vendor. Our reporting and analytics and generative engine optimization engagements build this stack for B2B SaaS clients in roughly that order, and Play 1 is always the one that changes the conversation.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.