Ask the same question five times, and the cited sources move
- Dimitris Drakatos
- 1 day ago
- 9 min read
Five registered questions, all built from one locked template, run through four provider APIs with web search enabled, five fresh calls per combination. That is 100 answers in 20 question-and-stack cells. Inside each cell I measured how much two repeats of that same question overlapped in the domains they cited, and the median of those 20 within-cell medians was 42.4%. Here is the method, the full table, and everything this does not prove.
What I found, in short
Median cited-domain overlap between repeats of the same question, measured inside each cell: 42.4%, taken as the median of the 20 within-cell medians.
14 of the 20 model-and-question cells had a median at or below 50%.
For the project-management question alone, cell medians ran from 23% to 67% depending on the stack.
Within one OpenAI configuration, photo books showed a 20% median overlap and email marketing showed 100%.
This measures repeatability of cited domains. It does not measure accuracy, source quality, recommendation quality, or how any consumer app behaves.
The question that started it
Ask an AI which project-management software suits a 20-person agency and you get five recommendations and a set of sources. Ask again, same wording, same API, same settings, and you get another five.
If you are tracking whether your brand turns up in AI answers, the useful question is whether those two source lists match. If they do not, a single measurement tells you less than it looks like it does. Worth knowing before anyone builds a reporting line on one query.
So I measured it.
What was asked
Five questions, registered before collection, all built from one locked template: What are the best [CATEGORY] for [CONTEXT]? Recommend five options and explain briefly why each is suitable. Include the sources you used. No provider-specific wording was added.
The five, verbatim, including the grammatical awkwardness of the first, which was preserved rather than tidied:
C01: What are the best project-management software for a 20-person agency? Recommend five options and explain briefly why each is suitable. Include the sources you used.
C03: What are the best email-marketing platforms for a small ecommerce brand? Recommend five options and explain briefly why each is suitable. Include the sources you used.
C07: What are the best language-learning apps for a beginner learning Spanish? Recommend five options and explain briefly why each is suitable. Include the sources you used.
C08: What are the best carry-on luggage brands for frequent travel within Europe? Recommend five options and explain briefly why each is suitable. Include the sources you used.
C12: What are the best photo-book services for someone in the UK? Recommend five options and explain briefly why each is suitable. Include the sources you used.
They cover B2B software, ecommerce tooling, consumer apps, physical products and a local service question. I am not claiming they represent any wider population of buying questions. Five is five.
How it was built
Five questions, four provider API stacks, five fresh calls per combination: 100 answers, 20 cells, 10 pairwise comparisons inside each cell.
Everything was collected on 4 September 2026 between 13:25:30 and 14:18:30 UTC, inside the registered collection window of 13:25 to 21:25 UTC, with provider, context and repetition order randomised from seed 20260907. All 100 responses were valid and search was observed in all 100. Preflight, failed and recovery runs are excluded.
The configurations, as recorded:
OpenAI Responses: configured gpt-5.5, returned gpt-5.5-2026-04-23, web_search, source metadata requested via web_search_call.action.sources.
Anthropic Messages: configured and returned claude-sonnet-5, web_search_20250305 named web_search, max_uses: 3, max_tokens: 1800.
Gemini generateContent: configured and returned gemini-3.6-flash, google_search: {}, temperature: 0.7.
Perplexity Agent API: configured and returned openai/gpt-5.6-sol, web_search.
Two things about that list matter. These are specific model-and-retrieval stacks recorded on one day, not the consumer ChatGPT, Claude, Gemini or Perplexity apps, not free tiers, not anyone's defaults. And Perplexity's identifier names an OpenAI model served through Perplexity's own orchestration, so no row separates model behaviour from the retrieval layer around it.
Generation settings were not matched across providers. The protocol asked for temperature 0.7 where the endpoint exposes it, and only Gemini received it explicitly. OpenAI's preflight returned an HTTP 400 saying temperature was unsupported, Anthropic's said it was deprecated for the selected model, and Perplexity's production request omitted it as well. Sampled response metadata reported OpenAI at temperature 1.0 and top_p 0.98, and Perplexity at temperature 1 and top_p 1, which is what came back rather than what was asked for. A disclosed difference between rows, not something the study controls.
What overlap means here
The unit is the set of unique registrable domains cited in an answer. Not URLs, not citation order, not repeat frequency, not the recommended products.
For two answers A and B, overlap is the number of domains they share divided by the number of distinct domains across both. That is Jaccard similarity.
A worked example with invented domains: if A cites a.com, b.com and c.com, and B cites b.com, c.com and d.com, they share two out of four distinct domains. Overlap is 50%.
Two clarifications, because both get mangled in practice. A pairwise overlap of 100% means those two domain sets were identical, not that the answers recommended the same five products. And a median of 42.4% does not mean 57.6% of sources changed, because the denominator is the union of both sets rather than the size of either one. If both answers in a pair cited nothing at all, that pair is recorded as missing rather than as perfect overlap.
Each cell holds ten pairwise comparisons, and they are not independent, since every answer appears in four of them. The cell figure is the median of its ten pairs, and the headline is the median of the 20 cell medians, not a pooled average.
The results
API stack | Project management | Email marketing | Language learning | Carry-on luggage | Photo books |
OpenAI | 67% | 100% | 83% | 33% | 20% |
Anthropic | 44% | 37% | 44% | 47% | 25% |
Gemini | 23% | 38% | 38% | 31% | 45% |
Perplexity Agent API | 62% | 57% | 57% | 40% | 40% |
Overall median: 42.4%. Fourteen of the twenty cells sit at or below 50%. Displayed values are rounded.
The project-management question alone runs from 23% to 67% depending on the stack, which is a wide spread for one question.
The contrast that stayed with me sits inside a single OpenAI configuration: 20% median overlap for photo books, 100% for email marketing. Same model, same settings, same afternoon.
One caution on that 100%. It is a median of ten pairwise values, so at least half those pairs were identical, not all of them. The cell's first quartile is 66.7%, so some pairs did differ. A cell median summarises; it does not describe every repeat.
The Anthropic token limit
The Anthropic row carries a limitation that belongs in front of you rather than in a footnote. Eleven of the 25 primary Anthropic responses stopped at the 1,800-token output limit. Fourteen finished normally.
So I ran a separate, separately approved collection: 25 Anthropic responses at a 4,096-token limit, nothing else changed. All 25 finished normally.
Context | Primary median | Separate higher-limit median |
Project management | 44% | 71% |
Email marketing | 37% | 63% |
Language learning | 44% | 50% |
Carry-on luggage | 47% | 41% |
Photo books | 25% | 43% |
Anthropic's median across contexts moves from 44% to 50%. Substituting that row into the 20-cell table as a post-hoc exercise gives 44.2% overall, with 12 of 20 cells at or below 50%. Recomputing that same substituted table, with only the replacement Anthropic row counted on provider-native citations rather than every extraction path the study normally uses, gives 47.7%. That third figure changes two things at once, the Anthropic row and the extraction scope, so it answers a narrower question than either of the others. Both are post-hoc sensitivities and neither replaces the primary result.
What I will not say: these are two separate collections of stochastic answers, not a controlled before-and-after on the same responses. The higher-limit run sits outside the primary evidence and does not replace it, and nothing here shows that truncation caused the differences. Carry-on luggage moved down rather than up, which is a fair reminder of how noisy five repeats can be.
How citations were counted
Answer citations only, not the wider pool of search results the model looked at. OpenAI provides url_citation annotations. Anthropic attaches citations to answer text blocks. Gemini exposes chunks referenced by grounding supports, where source titles were used to map Google redirect domains. Perplexity's are links in the final answer, and rows using that path are identified separately.
URLs were normalised with saved collector rules, documented tracking parameters removed, and the result deduplicated to registrable domains.
Two weaknesses worth stating. The Gemini title-based mapping does not verify where a redirect actually lands. And citation-set sizes differ substantially between stacks, which matters for a set-overlap metric: the primary pool holds 756 citations, split 127 OpenAI, 166 Anthropic, 272 Gemini and 191 Perplexity. Rows are not like-for-like.
The quality checks, and what they do not prove
The protocol required at least 98% extraction precision in a manual audit of 25 citations. I drew a pooled random sample of 25 from the 756 using seed 20260907, reviewed each against evidence cards, and recorded 25 correct decisions. That is 100% observed precision, and the gate passes.
It proves less than the number suggests. The sample was pooled rather than stratified, so it held 1 OpenAI, 3 Anthropic, 10 Gemini and 11 Perplexity citations, which gives almost no provider-specific assurance for OpenAI. It measures whether extracted citations were real, not whether any were missed, and it was not blind: AI assistants helped me navigate the evidence, though the verdicts were mine. No human audit was registered or carried out for the sensitivity run, and any such check would be extra quality assurance rather than a registered gate. Other AI systems reviewed the interpretation and re-ran mechanical checks, which is useful and is not independent validation or peer review.
What this does not tell you
Five questions, one collection window, one afternoon. Ten dependent pairs per cell. Uneven citation volumes and different extraction paths between rows. Different retrieval systems and unmatched generation settings. A known truncation issue on one row.
It says nothing about factual accuracy, source quality, recommendation quality, general brand visibility, or how the consumer apps and free tiers behave. It is not a ranking.
Two provenance points I would rather disclose than have found. The sensitivity run's approval was documented retrospectively and the original approval message's timestamp was not preserved. And a complete dollar cost for the primary run cannot be reconstructed from saved usage, since only Perplexity returned an exact total.
What I would do with this
Repeat the measurement before treating a domain's presence or absence in an AI answer as a result. One query is a sample of one, and on this evidence that sample moves.
What I cannot tell you is how many repeats are enough. This study does not establish it, and I would be wary of anyone giving you a number without showing their working.
The registered screens in the protocol label this an instability candidate. That is a descriptive screen defined in advance, not a verdict on any product.
Methods, evidence and corrections
Version: AISE-001, revision G copy, drafted 5 September 2026. Data: run AISE-001-prod-20260904-r1, collected 4 September 2026, analysis locked 5 September 2026. The human citation audit was completed on 5 September, which is why the graphic labels its date as collection rather than review.
The full methods note covers the locked protocol, the configuration snapshot, extraction rules and the sensitivity design. Download the evidence package (ZIP) if you would rather check the numbers than take them from me. It is AISE-001 public evidence v1: the original preregistered protocol, the cited-domain sets for all 125 responses across both runs, the cell-level result tables, the historical analysis functions, and an offline checker that recomputes the three headline results from those domain sets using nothing but the Python standard library. It starts at the exported domain sets rather than the raw answers, so it reproduces the metrics, not the citation extraction behind them.
SHA-256: f7a56d5c95d1527b31c2fcc63548b657c7f23f343c0be36a8050d1d90ad19d05
Disclosure: I own and run Wellknown, an AI-search audit service, so I have a commercial interest in this subject. This study measures citation repeatability across API stacks and does not test or establish anything about that service. No third-party sponsorship, funding or product execution role is asserted for this work.
If you spot an error, tell me through the contact form on this site and I will correct the article and note the change.
Comments