Prompt Tracking in LLMs for Brand Mentions

There is too much noise around prompt tracking.

Tracking tools - run prompts at scale using APIs, fan out prompts, and then run them. They use some normalisation methods to show the percentage visibility or whatever terms they have invented in their dashboards. Some also scrape actual ChatGPT chats instead of using the APIs.

Either way, there are gaps between what one sees in the tools versus what a user might actually see when prompting LLMs.

And there is a simple explanation for this – LLM outputs are probabilistic. In other words, they are non-deterministic. They cannot be fixed in time or beforehand. This is known as stochasticity.

In the case of LLM responses, stochasticity will show up as:

• Different wording for the same answer

• Different facts or hallucinations (relevant for product listings, as different products can show for the same prompts)

• Different tone and formality

• Different decision/reasoning chains

Example -Repeatedly asking a customer chatbot “How do I reset my password?” might produce a short step list once, and a longer, slightly different list next time– confusing for the customer and bad for analytics.

Stochasticity of LLM responses is influenced by various factors like model inputs (temperature, top-k/top-p), implementation/hardware details, tools usage/search grounding, and chat history.

The last part - chat history - is what could be deemed as LLM personalisation. In fact, if you are trying to check if your brand is visible/listed in a particular prompt, this becomes difficult to track.

Add to it the claim that, based on geography or IP, different products might be shown.

But the point here is – geo is a concern only if you are selling geo-sensitive products or services. If your product is geo-agnostic, the product lists in LLM responses shouldn’t change drastically. Unless new training data or search grounding indexes are used.

For example, the queries – ‘the best SEO tools in India’ vs ‘the best SEO tools in the US’ - eventually should have a very similar list. There isn’t any geography-specific feature in SEO tools that makes them geographically sensitive unless a product has limited geo services, geo-specific data, or language barriers.

If anything, this random recons might not be a great experience for LLM end users. What if they are not getting recommended the most suitable tools because of these issues?

In the example, LLMs should recommend the best tools out there for SEO, irrespective of whatever personal preferences LLMs have dug out for the person.

This is very similar to the challenge that Google faced/still faces for recommending product lists in SERP.

This is one of the reasons why LLMs always ground such prompt responses in search. Search indexes already have the lists ready-made. It just makes it easy for them to pick and display.