How to track brand mentions in large language models

Why a single cross-model number hides the thing you need

Ask five models the same question and you get five answers. That is not noise to be averaged away; it is the measurement. When your mention rate falls week on week, the only useful next question is which model moved, and a pooled figure has already destroyed the answer. One model dropping you while four held steady is a content or retrieval problem on that model’s inputs. All five drifting together is something else entirely, and usually something slower.

Pooling also quietly mixes measurements that are not the same kind of thing. In our own pipeline, three of the five models are handed web search results before they answer, one searches on its own, and one gets no web results at all. Averaging the last one in with the rest produces a number that is part “what is findable about you on the web today” and part “what the model already contains”, in unknown proportions. Those two should be read side by side, never summed.

So the working rule is simple: the model identifier is part of the measurement, not metadata about it. Every answer you store should carry the model string that produced it, and every chart should be splittable by that string.

What is actually measurable, and what no vendor publishes

You can measure outputs. Ask a fixed question, record the answer, record which model gave it, repeat on a schedule. From that you get rates you can defend: how often you are named, where in the list, alongside whom, and which URLs the answer cited. That is the whole evidence base, and it is enough to act on.

What you cannot get is a reason. Each vendor publishes a different amount about the stage that fetches web pages before a model answers, and all of it stops short of “why this brand was named”.

• Google describes “Retrieval-augmented generation (RAG)” as relying “on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index”, plus “Query fan-out: A set of concurrent, related queries generated by the model to request more information” (Google, AI features and your website, read 22 October 2026). That is Google describing generative AI features on Google Search. It is not a description of any other engine and must not be read as one.
• Microsoft frames the same stage as a different job from ranking: “Traditional search asks: which pages should a user visit? Grounding asks: what information can an AI system responsibly use to construct a response?”, and is explicit that “A common misconception is that grounding replaces search. It does not.” (Bing Search blog, read 22 October 2026). Scoped to Microsoft’s own stack.
• Perplexity has published an account of its retrieval stage: “Perplexity’s AI-native search stack retrieves and ranks candidate pages, surfacing the most relevant content for the model to ground its answers”, and “For each query, the retrieval and ranking engine searches a large, distributed index to select and order a small set of relevant pages within the search stack’s latency budget.” (Perplexity, Photon, read 22 October 2026). Note what it does not say: nothing about the criteria by which a retrieved page becomes a displayed citation.

Three vendors, three different accounts, all about their own surfaces, none of them a rule you can optimise against. Google puts the matter bluntly in the same guide we quoted above: “No third-party tool has access to our internal ranking or AI systems.” Hold every vendor in this category to that, including us. A monitoring tool is an instrument pointed at the output of a system, not a window into it.

The practical consequence is the reason this is a measurement problem rather than a research problem. None of the accounts above publishes the rule, so you watch the behaviour. We wrote up what a first pass at that looks like in our LLM brand audit guide, and the cross-engine comparison problem in brand tracking across ChatGPT, Gemini and Perplexity.

The series has an expiry date, and you have to track it per vendor

This is the part most measurement plans miss. The model behind a model string is not permanent, and a swap will show up in your chart as a visibility change when it is nothing of the kind.

OpenAI states it plainly: “As we launch safer and more capable models, we regularly retire older models. Software relying on OpenAI models may need occasional updates to keep working.” The documented minimum notice for generally available models is “At least 6 months”, shutdown dates are published per snapshot, and the status is machine readable: “To monitor model shutdowns programmatically, call the List models endpoint (GET /v1/models) and check each model’s optional shutdown_date field.” The same page records that “On June 11, 2026, we notified developers using older GPT-5 and o3 model snapshots of their deprecation and removal from the API on December 11, 2026”, listing gpt-5-nano-2025-08-07 with a shutdown date of 11 December 2026 and gpt-5.6-luna as the recommended replacement (OpenAI, Deprecations, read 22 October 2026).

Google documents the same hazard from the other end, in the naming. Gemini models “are available in either stable, preview, latest, or experimental versions”. A stable name “Points to a specific stable model. Stable models usually don’t change.” A latest alias “will get hot-swapped with every new release of a specific model variation”, and “For breaking changes, a 2-week notice will be provided through email before the version behind latest is changed.” (Gemini API, model versions, read 22 October 2026). Retirement is separate again: a deprecation is “the announcement that we no longer provide support for a model, and that it will be shut down in the near future”, and the published dates are “the earliest possible dates on which a model might be retired” (Gemini deprecations, read 22 October 2026).

Two things follow for anyone building a series. Pin exact model strings rather than floating aliases, so a change is something you do on a date you choose. And annotate the swap: when the model behind a question changes, the series before and after are two series. Reading the step as a drop in visibility is the most expensive mistake available here, because it sends a team to rewrite pages that were never the cause. We go into that failure mode at more length in measuring model evolution.

How Boldmention runs it across models

This is the pipeline, in the order it executes. Where a number appears, it is a number in the code.

Every active question goes out in parallel to the models your plan allows, drawn from a set of five model strings: gpt-5-nano (OpenAI), gemini-2.5-flash (Google), grok-3-mini (xAI), sonar (Perplexity) and deepseek-chat. We give you the strings rather than marketing names on purpose: a string is what we send, and it is the only part of this a vendor cannot quietly rename underneath us. The plan decides the pool, and that is enforced in the tracking function itself rather than in a pricing table: Starter and the free trial run gpt-5-nano, gemini-2.5-flash and deepseek-chat; Growth adds grok-3-mini; Pro adds sonar. One question against a Pro pool is five requests, not one.

How each model gets web results differs, by design:

• For gpt-5-nano, gemini-2.5-flash and grok-3-mini we call an external search API (Serper, with Brave as the fallback), take up to seven organic results for the question, and prepend them to the system message. The search is localised with the country and language stored on your company record. Within a run the three share one search call per question, so they are reading the same results, and Gemini’s own native search is switched off while we do it.
• sonar we send the question to without attaching anything. Perplexity’s own model documentation describes Sonar as “Perplexity’s grounded search model” (Perplexity, Models, read 22 October 2026).
• deepseek-chat gets no search results and no search tool. It answers from the model. Read it as a different measurement from the other four, not as a lower score.

Each answer is stored as its own row against the question and the model: the full response text, the model identifier, whether search was on for that call, whether your brand was mentioned, its position in the list, sentiment, a 0–100 prominence score, the competitor names found, and every URL in the answer with its domain. Brand detection is a word-boundary regular expression on your exact brand string, so a brand name sitting inside a longer word is not counted. If the brand is absent we store the answer with a score of 0 and make no further model call; if it is present, a second call (gpt-5-nano) produces the position, sentiment and prominence fields. Those three are one model judging another model’s answer, and we store them because they are consistent, not because they are ground truth.

Because the model identifier is on the row, the analytics view filters by model and compares models directly: average visibility, mention rate and the number of tracked answers, per model, over the window you choose. The prompt view lists the stored answers themselves with the same filter, which is where you go when a rate moves and you want to read what actually changed in the text. For a different angle on the same data, see share of prompt as the new share of voice.

What this does not do

• It does not cover engines outside those five models. No Claude, no Copilot, no Google AI Overviews.
• It calls provider APIs, not the consumer apps. An API model with search results attached is close to what a logged-in user sees in a chat app, and it is not the same product. Every vendor statement quoted above about model versions and retirement is about the API surface too. Ask any vendor in this category which one they measure.
• A failed model call leaves a gap, not a zero. We try a failing call up to three times; if a model still fails or times out, no row is written for that question and model. That is the honest behaviour, and it means a thin day for one model is a coverage problem to check rather than a visibility drop to explain.
• Competitor counting per answer is capped at ten. An answer naming more than ten of your stored competitors is truncated there, and that moves any share-of-voice figure in our favour. Worth knowing before you quote it.
• The scheduled repeat run excludes free-trial accounts. A trial gives you one snapshot at onboarding, not a series. Judge the product accordingly.
• It does not tell a paid placement from an organic mention, and it cannot tell you why a model answered as it did. Our pipeline records the text of an answer. Nothing in it identifies how a brand came to be there.

Where to start

If you are setting this up yourself, the order that saves the most rework is: write the category questions your buyers actually ask, fix their wording, decide which models you will run them against, and store the model identifier beside every answer from day one. Retrofitting that last one onto a year of pooled averages is not possible; the information was never recorded.

If you would rather have it run for you, request a free AI visibility report and you get the questions, the answers per model, and the brands named in them. For which models each plan covers and how the metrics are computed, see the frequently asked questions. For longer write-ups on share of voice, model drift and AI search visibility, see our blog. And if what you need first is to track brand mentions in large language models one engine at a time, start with the single-engine guide and its manual method.