Newsletter | Podcast | Blog | Insights

121WATT Online-Marketing-Blog
Zurück zum Blog
Blog GEO October 26, 2026 DEEN

Beyond Brand Mentions: A More Reliable Approach to Measuring GEO Visibility

How often does an AI assistant recommend a specific product when someone is ready to buy? A field report by Alexander Holl and Dr. Christoph Röck on our measurement approach in 121suite – and its limits.

The short version

Does visibility in AI assistants actually drive business? The industry is divided on this question. Studies focusing on clicks suggest the impact is minimal. We believe clicks are the wrong yardstick – but we don’t want to rely on assumptions either.

That is why we use 121suite to measure how often an AI assistant recommends a specific product when someone asks for a solution. Instead of simply counting brand mentions in individual answers, we simulate multi-step buying conversations. This gives us comparable signals on who makes the shortlist, who becomes the favourite and which criteria tip the balance.

It remains a measurement of synthetic conversations – not a picture of real buyers. And a higher recommendation rate is not yet proof of revenue. That is why we check the business effect separately, by asking customers. Below we show how we work, what we deliberately left out and where the limits are.

The debate: does GEO deliver anything at all?

Measuring visibility in AI systems is just as contested as GEO itself. In October 2026, Rand Fishkin summarised three studies that take a sceptical view of the business impact of AI visibility outside Google Search:

Fishkin himself draws a clear line: Google’s own AI (AI Overviews, AI Mode, Gemini) does matter. The opposing view is held by Michael King (“AI search is a branding channel”) and Christian Kunz, among others: a channel designed to make clicks unnecessary should not be judged by its clicks.

Fishkin’s whiteboard in detail. In his #5minutewhiteboard, Fishkin asks pointedly whether AI visibility is a boondoggle, a sham or a distraction. After reading the three studies, he says he is less confident than ever that more visibility in ChatGPT, Claude, Perplexity or Copilot brings much business for most companies. In the discussion he adds that, in clickstream data he is currently analysing, AI visibility correlates with sales far less than other channels do, with rare exceptions.

He puts three questions to the market:

  1. How many of the roughly one billion users ask search-like questions at all, the kind where they discover brands?
  2. How many of those questions fall into your own industry – and how strongly do the answers influence your customers?
  3. If both are unknown, why is so much money flowing into AI visibility tracking?

These questions are fair. For an individual company, though, the decisive question may not be how many questions are asked about its product – but whether its product is recommended when the question is asked. How important AI assistants have become, anyone can see in their own behaviour and in publicly available user numbers. That is why we developed an approach that lets us say how reliable a visibility measurement of AI product recommendations is. We see the measurement not as proof of business but as a diagnosis: who is recommended instead of us, by which criterion and based on which sources?

We check the business effect separately: we don’t ask in general about the “most important source of information”, but – right after a purchase – about what triggered that specific purchase, with randomised answer order and a dummy option. Biases remain all the same. That is why we read trends over time there, not absolute shares.

We believe both perspectives capture something important. Click studies measure demonstrable visitor flows. Our measurement checks whether an offer is recommended in a simulated selection process. Neither alone answers the question of the actual contribution to revenue.

Critical objection Our answer What remains open
“AI brings hardly any traffic.” We measure the recommendation before a possible click. Whether it leads to additional revenue – for that, we ask customers.
“Synthetic prompts are not buyers.” Multi-step dialogues, checked prompts, repetitions, web search in the target market. Real questions, personalisation and user context.
“Google matters more.” Gemini is one of the four assistants we measure. We do not measure AI Overviews and AI Mode directly.
“GEO tricks only work short-term.” We show criteria and sources, i.e. starting points for brand and PR work. The effect and durability of individual measures.
“A single global number hides lost markets.” We measure per market, in its language and with its web search. –

Why classic share-of-voice measurement isn’t enough for us

In classic Google Search we know rankings: for a keyword, a page holds a certain position. It can change, but it is basically observable. ChatGPT and other AI assistants work differently. Every answer depends on the prompt, the conversation so far, the model used and, where applicable, the web search.

Many GEO tools therefore ask a list of questions and count how often a brand appears. This has some value, but it mainly answers the question: “Does the AI know our brand?” For marketing and sales, however, we care about something else:

Is our specific product recommended when a potential customer is preparing a decision – and if not, which competitor is recommended instead?

From this we derive three analytical questions: How visible is our offer during the selection? Which characteristics get it recommended or passed over? And which sources does the assistant use for its answer?

We measure products and services, not just brands

An example from professional training: a provider may be recommended regularly for an SEO seminar, yet barely appear for an AI training course for teams. A single brand-level score would obscure this distinction. That is why we measure every product and service individually.

The starting point is the respective product page. From it we take the name, target group and benefit, as well as typical problems and expected gains (pains and gains). The page is read once; the stored version is kept for drafting prompts until someone deliberately updates it. AI suggestions are only adopted after human confirmation.

From first interest to a concrete recommendation: simulated buying conversations

Instead of asking the assistant once for the “best providers”, we hold a multi-step conversation with it. We start openly, have it form a top 3 and then ask for a favourite. After that we change one criterion on purpose, such as price or practical relevance. Every answer stays in the conversation context – just like in a real dialogue. One unit of measurement is one conversation: prompt × assistant × repetition.

Flow of a simulated buying conversation: open question, top 3, favourite, criterion

Fig. 1: Four stages that build on each other; follow-up questions run in separate conversation branches.

Stage Example question Weight
Open question “SEO training specifically for online shops?” 1
Top 3 “Which three providers would you recommend?” 2
Favourite “Which one would you choose?” 3
Favourite by criterion “And if price matters most to me?” 3

We deliberately keep two kinds of follow-up questions separate from the main conversation. First, we ask about additional criteria each time from the same conversation state. Otherwise the price question could distort the later quality answer. Second, if our brand is not mentioned, we ask why – but again only in a branch. If we introduced our brand directly into the main conversation, the follow-up question would influence the next recommendation, and in the end we would be measuring our own question.

At the end, we have the assistant summarise its conversation in a table. This self-report by the model is a control value for us. What counts are the answers we evaluate, not what the model claims about itself afterwards.

How we develop and check the prompts

The reliability of our measurement depends heavily on the quality of the prompts. If we choose poor or very similar prompts, even the most precise index only measures the weaknesses of our questionnaire. That is why we don’t simply let the prompts be generated freely.

We use established marketing frameworks: the funnel (TOFU, MOFU, BOFU), AIDA, See-Think-Do-Care, the Messy Middle, as well as buying motives or pains and gains. Custom frameworks are possible too. Stages that don’t fit are deliberately excluded.

Every prompt must be phrased generically and aim at a provider recommendation. It may name neither our brand nor a competitor. A question like “What alternatives are there to provider X?” would be a different kind of measurement and does not belong in this panel.

With the prompts we pursue two goals that only seem to contradict each other: within a stage, the questions should be as different as possible – and yet all fit that very stage as well as possible. We want to capture different paths to the same buying intent, not ask the same question six times.

Prompt check: draft, blind check, redraft, duplicate check

Fig. 2: Draft, blind check, at most one redraft and a duplicate check before measuring.

Is a checked prompt set meaningful for all conceivable prompts?

The basic problem of every GEO measurement: for a topic like “SEO seminar” there are not ten questions but millions of different conversations. People choose different words, bring different prior knowledge, describe different situations and ask follow-ups differently. No measurement can hold all these conversations, and nobody knows their actual distribution.

So we proceed as in empirical research: we draw a sample. It consists of prompts on the same topic that are as independent of each other as possible – different motives, question forms and needs, no two variants of the same question. Independent here means: the prompts share the topic but not the wording. The next two sections describe how we ensure this.

This leads us to a key principle:

  • When an AI assistant consistently recommends the same provider across a diverse set of prompts, this provides evidence that the recommendation is robust to variations in wording and user intent. However, without representative data on real-world queries, we cannot reliably estimate how frequently the provider would be recommended across all possible user interactions.
  • When different providers are named for each prompt, the result is not stable. A single rate then has limited explanatory value – or the AI has not yet formed an “opinion” on the topic. This is a finding in itself: the market is still open in AI, and small changes to the question determine the recommendation.

The principle is familiar from survey research: if ten very different people independently say the same thing, it is likely that others share this view. If everyone says something different, there is no settled majority opinion.

To separate wording from chance, we combine two comparisons: the agreement between repetitions of the same prompt and the agreement between different prompts for the same product (calculation in the section on stability further down).

Repetitions of a prompt Different prompts Reading
agree agree Robust recommendation: probably also holds for many unmeasured variants.
agree differ The wording decides: each question has its own stable answer. The result only applies to questions of this kind.
differ differ No settled “opinion”: chance dominates. More repetitions help, or the market is still open in AI.

It is important to put this in context: our prompt sets resemble a quota sample. We stratify by buying stages or motives and fill each stratum with questions that are as different as possible. Within the strata, however, we do not draw at random from real conversations, and nobody knows how large each stratum is in reality. The inference to unmeasured prompts is therefore a plausibility argument, not a statistical proof. How far it carries depends on how different the prompts really are – which is exactly why we check variety and duplicates so strictly. We deliberately haven’t set a fixed threshold above which a set counts as meaningful yet; we read the agreement as a continuum.

How we measure stage fit and separation

When drafting, every slot in a stage is given a question form and a need that fits the motive. A second model then assigns the prompts blind to all stages of the framework – including the category “none”. The intended stage is not revealed. For each prompt it distributes points according to how strongly the individual buying motives come through.

From this we calculate two indicators:

Formula: fit and separation

Here, a is a stage’s share of the points. Fit shows how clearly a prompt expresses its motive. Separation shows how clearly it stands apart from the strongest neighbouring motive. If it is too low, the prompt measures the neighbouring motive just as well.

If a prompt falls below our minimum or does not aim at a concrete provider recommendation, we redraft it once with targeted feedback. This second check result is documented as well – so we don’t optimise endlessly for a test score. If a prompt fits another stage better, it is moved there.

What we observed: for 192 motive prompts for 121WATT seminars, the intended stage received the highest score in 96% of cases. On average, however, only about two thirds of the points went to the prompt’s own stage. So the questions can mostly be assigned well, but often contain aspects of neighbouring motives.

How we measure variety within a stage

Content duplicates. A checking model groups, across the whole question set, the prompts that ask for the same thing – even if they look different linguistically. From each group we keep the prompt with the better stage fit; the others are redrafted.

Formal variety. Each question gets a form signature made of sentence form, context and comparison mode. For each stage we count the distinct questions and the distinct form signatures. Ideally, both equal the number of prompts used.

Why we set forms and needs as early as the draft: without a prescribed form, 73% of motive prompts followed practically the same pattern (“I’m looking for …, can anyone recommend something?”). The needs must fit the stage’s motive. Otherwise a need from another motive pulls the prompt towards that motive.

We also tested embedding similarity. It proved unsuitable: it mainly detects the shared product vocabulary, not the buying motive. That is why we check fit and duplicates through a model’s judgement of the content.

Interim conclusion: good prompts are close in content to their stage, but as little redundant among themselves as possible. These checks improve our constructed panel; they do not guarantee a representative picture of real user questions. And once a prompt set has been measured, we don’t change it quietly: new prompts are a new sample, and that break is marked in the time series.

The conditions under which we measure

For every product and every prompt, 121suite holds separate conversations with ChatGPT, Gemini, Claude and Perplexity via their APIs – each with web search.

Language and market are decisive. Anyone analysing the Spanish market must ask in Spanish and measure in Spanish web search – regardless of whether the user interface is in German. Without a location setting, some providers search from the US; with one, the share of local sources rises noticeably. Not every interface offers this control, so the assistants are not fully comparable.

Repetition instead of false precision. The answers of generative models fluctuate. That is why we repeat every conversation several times. This shows whether a provider is named regularly or only turns up by chance. Measurements run manually or monthly. Interrupted individual conversations are re-run with the same model so that two points in time don’t get mixed.

The model is part of the measurement conditions. By default we use the most capable model of each provider whose web search we have tested. If it is replaced, we document the model change in the time series. After all, a different model is a different measuring instrument.

When does a mention count as a recommendation?

A mere mention is not the same as a recommendation. That is why a separate evaluation model analyses every answer, extracts all providers named and assigns each mention a role. We also match our own brand against the raw text, so that a missed extraction doesn’t simply get lost.

Role Meaning
Recommendation The assistant explicitly advises in favour of a provider.
Neutral The provider is mentioned but not assessed.
Critical The provider is mentioned with a warning or caveat.
Source only Our own website is cited, but our brand does not appear in the answer at all: the content is read, the offer is not recommended.

If several roles occur for the same provider in one answer, the critical mention takes precedence over a recommendation and a neutral mention. A warning should not disappear into a positive average.

There is also a practical problem: one assistant writes “Haufe”, another “Haufe Akademie”. The application suggests possible merges, but a human has to decide. The raw answer stays unchanged; assignments can be undone at any time.

The visibility index: why a favourite counts more than a mention

First, for each conversation stage we calculate the share of answers in which a provider appears. From this we form a weighted average between 0 and 100. The open question has weight 1, the top 3 weight 2, and both favourite stages weight 3 each. Later stages weigh more because they come closer to a concrete decision.

Formula: visibility index

Stages without answers drop out of the denominator and are not mistakenly counted as zero. Across several products, every product counts equally: we aggregate per product first and then take the average. Otherwise products with many prompts would distort the overall picture. A product without a measurement appears as “not measured”, never as 0%.

One conceptual clarification matters to us here: the index is a weighted indicator of recommendations and mentions. It is not a probability that a real buyer will receive a particular recommendation. Its sensible use lies in structured comparisons – with competitors, between assistants and over time. We describe when a lead is robust in the section on the reliability of the measurement.

121suite test run: overview of visibility, favourite, top 3 and mentions
121suite test run: provider ranking

Fig. 3: Excerpt from a 121suite test run – visibility. Overview of visibility, favourite, top 3 and mentions compared with the strongest competitor, followed by the ranking of providers (based on 787 conversations). Screenshots from the German interface.

How reliable is the measurement – and what can’t it do?

We look at two questions separately. Reliability: do we get similar answers under similar conditions? Validity: are we measuring the behaviour we are interested in at all?

What we can back up statistically

Every rate is a share p out of n answers. As a simple approximation we use the binomial standard error:

Formula: standard error of a rate

For the visibility index we combine the standard errors of the stages according to their weights, treating the stages as independent. When comparing with a competitor, we test the difference d two-sided at the 95% level:

Formula: standard error of the difference and the 1.96 threshold

Only then does the analysis show “ahead” or “behind”, otherwise “level”. An example: with 300 answers and a favourite rate of 43% versus 45%, the threshold is around 7.9 percentage points – a difference of two points is “level”. Important: for us, “level” explicitly means “no statistically detectable difference”, not that two providers are really equally strong. The same rule applies to each AI individually, per product and per criterion.

This appears to be a clear threshold, but it requires an important caveat: the calculation treats answers as independent. In fact we repeat the same prompts, and the stages of a buying conversation build on each other. 30 similar answers are not automatically 30 independent observations. The calculated standard error may therefore be too small, and a reported lead may look more certain than it is. For now, the 95% threshold is a guide under simplifying assumptions, not proof.

A sensible next step is a bootstrap at prompt level: whole prompts, including all repetitions and conversation stages, are resampled instead of treating individual answers as independent. This allows a more robust estimate of the uncertainty for the index and for differences. It does not, however, solve the question of how representative synthetic questions are of real buying conversations.

Stability: why the variety of prompts is decisive

If an assistant names the same provider ten times for ten almost identical questions, that is at first only a stable answer to a very similar question. But if it recommends that provider across different, stage-consistent questions and repeated conversations, the recommendation evidently depends less on any single wording.

For top-3 lists we measure agreement with the Jaccard coefficient: shared providers divided by all providers named. {A, B, C} and {A, B, D} give 2/4 = 0.5; identical lists give 1. We calculate it separately:

  • between repetitions of the same prompt with the same assistant,
  • between different prompts for the same product with the same assistant,
  • per provider: in how many prompts it is in the top 3, with how many assistants in at least half of the conversations, and whether it is named in every repetition or only occasionally.

121suite test run: agreement of the top 3 across repetitions and prompts
121suite test run: stability per provider

Fig. 4: Excerpt from a 121suite test run – stability of results. Agreement of the top 3 across repetitions and across prompts, per assistant, followed by the classification per provider. Screenshots from the German interface (“stabil” = stable, “wackelig” = shaky, “zufällig” = random).

In the example, the top 3 agree 38% of the time across repetitions, but only 22% across different prompts. Prompt wording therefore has a substantial impact on the results. Even so, the leading provider is in the top 3 for 106 of 114 prompts – exactly the kind of recommendation that, in line with the key principle above, indicates robustness to variations in wording.

Our central hypothesis: the more different the questions within a buying motive and the more consistent the recommendations, the more robust the statement about which provider an AI prefers under the tested conditions. Analysing agreement per framework stage is the obvious next step.

This allows a well-founded statement about recommendation stability within the panel examined. It does not allow a statement like “Every real user has an 85% probability of being recommended this provider”. For that we would need to know how real users distribute their questions and which contexts they bring with them.

What uncertainty remains

  1. Constructed prompts. They are not a representative sample of real user queries. Reliable volumes of real buying prompts do not exist yet.
  2. API is not the consumer app. System instructions, personalisation, memory and location may differ. We model a neutral user without history.
  3. Model changes alter the measuring instrument. A jump in the index can have technical rather than market causes.
  4. Coding by a model. The comparison with the conversation summary reveals discrepancies but does not prevent misclassifications.
  5. Sample size. Four assistants, several repetitions and a dozen prompts add up to well over a hundred conversations per product. That reveals clear differences, not arbitrarily small ones.
  6. Many comparisons. Providers × AIs × products add up to many tests. If there is truly no difference, roughly one in twenty will appear as “ahead” or “behind” by chance. Individual cells should be read with caution.
  7. Averaged rates, pooled case numbers. We average the rates across products, while n is the total number of answers – an approximation.
  8. Token consumption and cost. Every conversation consists of several questions with web search, and the history grows with every stage. Web search in particular drives up the number of tokens processed. On top come the calls for evaluation and prompt checks. Costs increase linearly with the number of prompts, assistants and repetitions, while statistical precision improves much more slowly: halving the standard error takes four times as many conversations. Every measurement is therefore a trade-off between precision and budget.

Our interim conclusion: stage fit, variety, repetitions and stability analyses make the measurement more transparent and more robust. We still show the statistical uncertainty explicitly – rather than turning a synthetic panel into certainty about the behaviour of all buyers.

Does GEO visibility bring business? We ask our customers

The visibility index tells you whether an assistant recommends you. It does not tell you whether that creates business. For that, we rely on an instrument we have used for years: customers’ self-reports right after the purchase.

Why not via click data? An AI recommendation leaves hardly any trace of its own in web analytics. Someone who reads about a provider in ChatGPT later googles it or types in the address directly. In web analytics this shows up as a brand search, an ad click or direct traffic – exactly the channels that already bundle many other causes. We showed this back in 2019: in our article on measuring word-of-mouth and cross-channel effects (in German), 74% of the seminar registrations that web analytics counted as “direct” were, according to customers, due to existing customers and recommendations. Even for registrations via paid Google ads, only 41% named Google as the actual trigger. Web analytics shows where someone comes from, not why.

How we ask. We apply this approach to GEO:

  1. Timing: right after the purchase, on the thank-you or confirmation page. In B2B also in the registration form or the first sales conversation.
  2. Question: about the trigger, not the channel – roughly “What brought you to us?”.
  3. Answers: closed options that fit the business model, with a separate option “Recommendation from an AI assistant (e.g. ChatGPT, Gemini, Claude, Perplexity)” kept apart from “Google search” and “Recommendation from colleagues or friends”.
  4. Reducing bias: the order of the options is randomised on every call, plus an open option “something else” and a dummy option that nobody can seriously choose. Anyone who clicks it is clicking at random – we filter out those answers.

How we link it to the index. We track the share “AI assistant” month by month, per product where possible, and set it next to the visibility index from the same month. If both rise together, that is a hint, not proof of causality. Anyone who evaluates a measure after a few days measures fluctuations rather than effects; the monthly cycle therefore suits both measurements.

Limits of self-reports. They only capture conscious, remembered influences. When purchase volumes are low, the sample size may be insufficient. And people like to answer in a way that sounds smart – which is exactly Rand Fishkin’s main objection to surveys – so “ChatGPT” may be over- or under-reported in surveys. Randomised order, the dummy option and the open answers help to detect such effects. That is why we mainly evaluate how the share develops over the months, not its absolute level. Even so, self-reports are currently the most direct way to see the business effect of an AI recommendation at all.

Our conclusion

GEO visibility is not a fixed Google position. It changes with the question, the course of the conversation and the model. For a sensible measurement we therefore need not one prompt and a brand count, but a transparent sample, standardised conversation stages, repetitions and clear evaluation rules.

Our most important decisions were:

  1. Measure products, not just brands.
  2. Analyse buying conversations instead of isolated single questions.
  3. Check prompts systematically for fit and variety before using them.
  4. Always look at recommendations together with their sources and selection criteria.
  5. Don’t derive the business effect from clicks – ask customers.

This makes it much easier to see where offers stand in an AI-supported selection process and which measures might pay off. However, these measurements should not be interpreted as definitive evidence of real-world user behaviour, and they are no substitute for business figures. Whether GEO is a distraction is ultimately decided not by traffic, but by customers’ answer to the question of what brought them to us.

We welcome your perspective: how do you measure recommendations in AI assistants? And where do you see the biggest methodological limits? We look forward to your experiences.

About 121suite and 121 Apps GmbH

121suite.com (currently in closed beta) is a marketing platform that brings SEO, paid and GEO data together in one application. At its core is the measurement described here of how visible products and services are in ChatGPT, Gemini, Claude and Perplexity: simulated buying conversations, checked prompts, source and competitor analysis, each in the language and with the web search of the target market.

We develop 121suite at 121 Apps GmbH, together with other products on topics that keep us busy:

  • 121surveys.com – online surveys (launched)
  • paperball.finance – ETF backtests, Monte Carlo simulations and securities watchlists (launched), part of the paperball.de product family
  • 121suite.com – marketing platform with GEO visibility measurement (currently in closed beta)
  • 121assist.com – AI-powered RAG chat systems for customer support and knowledge transfer (currently in closed beta)

More about us and all products: 121apps.com

Sources

1 Klick genügt

Wie hilfreich war dieser Artikel für dich?

Deine Stimme entscheidet mit, welche Themen wir als Nächstes vertiefen.

Diskutiere mit uns das Thema:

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

▲