Newshot

Ask AI the same question twice and 64% of answers name different brands

Original text(via HN AI)

Yesterday we posted that AI answers change from one day to the next. Readers on Hacker News and Reddit asked good questions:

  • Is it the day, or would the answer change anyway? We only asked once a day, so we couldn't tell.
  • Is "changed" too blunt? Swapping one brand out of six isn't a new answer.
  • Where are the statistics?
  • Wouldn't a more specific question get a stable answer?

So today we tested each one. We ran the same 210 questions two more times, at the same moment. Then we fetched and read the pages every answer cited, to see what decides which brands get named.

The short version

  • The answer changes even when you ask twice at once. 64% of answers named a different set of brands with no time in between. A day apart, it was about 80%. So most of the change isn't about time. It happens on every ask.
  • Most brands are named only some of the time. Over five runs of the same question, 62% of the brands AI named were missing from at least one answer. Only the top two or three brands were named almost every time.
  • The pages AI reads decide who gets named. Brands mentioned on 3 in 4 of the cited pages were named in 70% of answers. Brands on fewer than 1 in 10 were named in 3%. The same brand was named twice as often on questions where more of the pages mentioned it.
  • Specific questions are less stable, not more. Few pages answer them, so each run finds different ones.
  • ChatGPT changes about as much a minute apart as a day apart. Google's AI changes more each day.

If you sell something, one check of "does ChatGPT recommend us?" tells you very little. What matters is how often you're named across many answers. And that depends mostly on how many of the pages AI reads mention you.

How we tested it

We already ask 210 buying questions every day, across 21 categories, on ChatGPT, Gemini, Google AI Overviews and Google AI Mode. On 19 November we added two runs, A and B, sent at the same moment.

RunWhen (UTC)Answers
Day 117 Nov, 12:33839
Day 218 Nov, 00:00840
Day 319 Nov, 00:00836
Run A19 Nov, 17:02–17:30996
Run B19 Nov, 17:02–17:30, alongside A999

Runs A and B also had 42 extra, very specific questions (more on those below). That gave us two answers to the same question on the same engine at five gaps: no gap, 12 hours, 17 hours, 24 hours and 36 hours.

For each pair of answers we measured:

  • Brand overlap: brands named in both answers, divided by brands named in either. 100% means the same brands.
  • Rank-biased overlap (RBO): a standard measure for ranked lists that counts the top of the list more than the bottom (Webber et al., 2010). 1 means the same ranking.
  • Unchanged: both answers named exactly the same brands.
  • Page overlap: the same as brand overlap, but for the pages each answer cited.

Every answer comes from a new session with no account or history, from the US, collected through DataForSEO. The 95% intervals resample questions, not answers, so four answers to one question never count as four separate results.

A correction. In the day-2 post we said 84% of answers changed overnight. That number came from our app's automatic brand matching, which also counts some names that aren't recommendations. Recounted with the study's published brand list (brands.json), it's 75%, and brand overlap was 68%, not 61%. Same direction, smaller number. We've added a note to that post. Every number here uses the published list and scripts.

Finding 1: the answer changes even when you ask twice at once

图表:Answers that named a different set of brands, by time between the two runs

Same 210 questions on all four engines, about 820 pairs of answers per row. The thin band is the 95% interval (resampling questions).

Gap between runsBrand overlapRBOUnchangedSame #1 brand
None (same moment)73% (71–76)0.8436%77%
12 hours68% (65–70)0.8025%69%
17 hours68% (66–71)0.8028%73%
24 hours64% (62–66)0.7819%68%
36 hours64% (62–67)0.7820%67%

Ask the same question twice at once, and the typical answer still swaps about one brand in six. Ask a day later, and it swaps a bit more. That extra change is real (5 to 10 points of brand overlap, p < 0.001 against every other gap), but it's small. About 80% of the difference between two answers a day apart is already there when you ask twice at once.

The date doesn't matter either. Two runs 17 hours apart on the same day were no more alike than two runs 12 hours apart on different days.

Finding 2: most brands are named only some of the time

We asked each question five times on each engine. A typical answer named 4.9 brands. But across the five answers, 7.5 different brands came up, and only 2.9 were named every time.

Of all the brands named for a question on an engine:

  • 38% were named in all five answers.
  • 39% were named in two to four.
  • 22% were named only once.

Position matters. Chance a brand was named again when we asked a second time at the same moment:

Position in the first answerNamed again
#195%
#289%
#384%
#477%
#574%
#6 or lower66%

So for most brands, "are we recommended?" has no yes-or-no answer. A brand named in 3 of 5 answers is missing from 2 in every 5 checks. Check once, and you might think you're in when you're usually out, or the other way round.

Finding 3: the pages AI reads decide who gets named

Why would the same question get different brands a minute apart? ChatGPT, Gemini and Google's AI all search the web before they answer, and each answer lists the pages it used. So we fetched those pages. We could read 9,077 of the 11,897 that were cited; the rest blocked us or weren't web pages.

For each question on each engine, we took every page cited in any of the five runs and counted how many of them mention each brand. Then we compared that with how often the brand was named.

图表:How often a brand was named, by how many of the cited pages mention it

Share of the five answers that named the brand. Each brand is counted once per question and engine. Pages: every page cited for that question in any of the five runs, 9,077 read in all, without the brand's own website.

Share of cited pages that mention the brandNamed in (share of answers)Named every timeNever named
Under 10%3%0%92%
10–25%8%2%83%
25–50%21%10%64%
50–75%48%31%34%
75% or more70%54%16%

The more of the pages mention you, the more often you're named. The jump comes at about half: brands on fewer than half the pages were named in about a fifth of answers or less.

Two more numbers:

  • 95% of named brands appeared on at least one cited page. Very few brands were named without a page behind them.
  • 74% of brands named every time were on at least half the pages.

Isn't that just big brands being on more pages and named more? We checked by comparing each brand with itself. For 642 brands that appeared on more than one question or engine, the same brand was named in 30% of answers where it was on more of the pages, and 14% where it was on fewer (difference 14 to 17 points, 95% interval). Being a big brand helps. Being on the pages helps on its own.

It holds on every engine. For brands on 3 in 4 of the pages: ChatGPT named them 88% of the time, Gemini 74%, AI Overviews 65% and AI Mode 59%.

This also explains why answers change at the same moment. Each run picks a different handful of pages. When two runs cited exactly the same pages, 79% named exactly the same brands. When they had no pages in common, 5% did.

All pairs of runs pooled: 4,112 pairs of answers. Pages are matched by URL, ignoring tracking parameters.

The model's own wording matters too: even with identical pages, one answer in five changed its list. But change the pages and almost every answer changes.

This matches our first study, where 90% of recommended brands were on a "best X" list the engine had just read. A brand on most of the pages shows up whichever ones the engine picks. A brand on a few shows up only when those few get picked.

Finding 4: specific questions are less stable, not more

Several readers said a more specific question would get a steady answer. So we wrote 42 of them, two per category. For example:

  • best CRM for a 5-person real estate team under $30 per user per month that syncs with Gmail and has a mobile app
  • best mattress under $1,200 for a side sleeper with lower back pain who sleeps hot
  • best travel insurance for a 2-week trip to Japan for a 68-year-old with a pre-existing heart condition

We asked each one twice at the same moment and compared them with the broad questions from the same runs.

Asked twice at onceBroad questionsSpecific questions
Brand overlap73% (71–76)56% (51–60)
Unchanged36%12%
Same #1 brand77%56%
Page overlap54%19%
Brands per answer5.13.5

Specific answers name fewer brands, and a short list changes more when one name changes. So we also compared answers of the same length. With 4 to 6 brands, specific questions kept 57% of their brands against 75% for broad ones. They were less stable on every engine.

The reason is the pages. There are dozens of "best CRM" lists, and they all name the same leaders. There are very few pages about CRMs for 5-person real estate teams on Gmail, so each run finds a different few.

We wrote these 42 questions ourselves, so treat the exact numbers with care. The direction was clear, and the intervals don't overlap.

Finding 5: ChatGPT changes on every ask, Google's AI changes over days

Brand overlap by engine:

EngineSame moment12 hours24 hours36 hours
ChatGPT76%74%72%76%
Gemini72%64%61%64%
AI Overviews77%69%63%62%
AI Mode69%62%60%56%

ChatGPT showed no steady change over time. Answers 36 hours apart were as alike as answers asked at once (difference 0.9 points, interval −2.3 to +4.1). Its answers change about as much on every ask as they do over a day and a half.

Google's AI changed more each day. AI Mode went from 69% overlap at the same moment to 56% after 36 hours, and the pages it cited moved with it: page overlap fell from 57% to 29%. Both Google engines build answers from Google's own search results, which likely explains it.

What to do with this

Don't trust a single check. Not a screenshot, not one run of a tool, not one day. Ask each question several times on each engine and count how often you're named. That share is your real number.

Find the pages AI reads for your buyers' questions. Then count how many of them mention you. The more do, the more often you're named. On under a quarter of the pages, you'll rarely be named. On about half, you'll be named about half the time. On three quarters or more, most of the time.

Get onto the pages that leave you out. Start with the ones cited most often. Many are "best X" lists, reviews and comparison pages. Ask to be added, offer a demo or a sample, or publish your own honest comparison.

Write pages for the specific questions. So few pages answer them that the engine has little to pick from. One good page about your exact use case can be the page it finds.

Check Google's AI more often than ChatGPT. ChatGPT moves about the same amount on every ask, so weekly averages are enough. Google's AI drifts within a day or two.

What we can't tell you yet

  • One country, one language. Every answer is from the US, in English.
  • Logged-out only. We see what a new visitor sees. Someone logged in, with history, may get something else. We can't set the temperature in these apps, and neither can the people using them.
  • No runs without web search yet. Asking with search turned off would show what the models recommend from memory. It's next.
  • Pages fetched today. We read the cited pages on 19 November, so a page may have changed since the answer cited it. A quarter of pages couldn't be read and are left out.
  • Brand matching is automatic. It uses a fixed list of about 700 brands. A brand outside the list isn't counted, which matters most for the specific questions.
  • Three days so far. The study runs until 23 November, and we'll update this post.

Method and data

  • Questions: 210 buying questions across 21 categories, plus 42 specific questions (2 per category) in runs A and B.
  • Engines: ChatGPT, Gemini, Google AI Overviews and Google AI Mode. US, English, new sessions, collected through DataForSEO.
  • Answers: 4,510. Pairs compare the same question on the same engine in two runs. Pairs where neither answer named a brand from the list are left out.
  • Pages: 11,897 cited pages, 9,077 read. For each brand, pages on its own website are left out.
  • Statistics: 95% intervals from 2,000 bootstrap samples of questions (of brands, for the same-brand test). Differences between gaps are paired by question. RBO uses p = 0.9, extrapolated.
  • stability-pairs.csv: every pair of answers, with both brand lists and every measure.
  • page-coverage.csv: every brand on every question and engine, with how many runs named it and how many cited pages mention it.
  • stability-results.json and coverage-results.json: every number in this post.
  • stability.py and coverage.py: the scripts that produce them.

Disclosure: I run Supercite, which tracks AI recommendations for brands. The study runs on the same system, on its own account.

FAQ

Why does ChatGPT give different answers to the same question?

Because it searches the web before it answers, and each time it picks a different handful of pages. In our test, 64% of answers to the same buying question named a different set of brands when asked twice at the same moment. When two answers cited exactly the same pages, 79% named the same brands.

Do AI recommendations change over time?

On Google AI Overviews, Google AI Mode and Gemini, yes: answers drift further apart each day, along with the pages they cite. On ChatGPT we saw no steady change over 36 hours beyond the change you get by asking twice.

How do I get AI to recommend my brand more often?

Be on more of the pages it reads. Brands mentioned on 3 in 4 of the pages cited for a question were named in 70% of answers. Brands on fewer than 1 in 10 were named in 3%. Find the pages cited for your buyers' questions and get onto the ones that leave you out.

Are specific questions more consistent than broad ones?

No. Our 42 specific questions kept 56% of their brands when asked twice at once, against 73% for broad questions, because few pages answer them and each run finds different ones.

How many times should I check whether AI recommends my brand?

Enough to see a share, not a yes or no. 62% of the brands AI named for a question were missing from at least one of five answers. Ask each question several times on each engine and track how often you're named over time.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗