I asked four AI models their favourite colour 50 times each: guess how many of the 200 answers differed
Experiment ID: AISE-B01. Run ID: AISE-B01-prod-20260908-r2. Part of my AI Search Experiments series.
Zero. Four models, 50 answers each, 200 responses in total, and not one of them differed from the others.
Every one of the 200 answers was blue.
The short version
One question, asked 50 times each to four models: OpenAI's gpt-5.6-luna, Anthropic's claude-sonnet-5, Google's gemini-3.6-flash and Perplexity's perplexity/sonar. 200 responses in total.
I called the models directly through their developer APIs. This was not ChatGPT, Claude, Gemini or Perplexity, the consumer apps, and it does not tell you how those would answer.
Every response was exactly blue. All 200 were valid against a 148-word colour list fixed before collection.
Entropy came out at 0 bits per model, which is a description of this sample: no variation was observed in these 200 responses.
The run took 4 minutes and 24 seconds, from 08:50:41 to 08:55:04 UTC on 8 September 2026. Locally calculated cost: under one US cent.
The result records what came back. It does not explain why, and I am not going to pretend it does.
I registered a 148-word colour list before collection so the parser had somewhere to put whatever came back, everything from aliceblue to yellowgreen. 147 of those words never appeared.
The normalisation rules went the same way. Strip a trailing full stop. Reject an answer that turns up wrapped in a sentence. Catch a response that names two colours instead of one. Treat a refusal as invalid. Not one of them changed a single response, because every answer arrived as the bare word with nothing to tidy.
Two hundred responses. One word. Four minutes and 24 seconds. Under one US cent, locally calculated.
A result with no variation at all warrants checking your own collection and extraction before you believe it, so that is what happened next. The checks are below the results, along with the parts of the question they cannot settle.
What exactly did I ask?
One prompt, byte-for-byte identical on all 200 requests, with no system prompt, no examples and no conversation history supplied by me:
What is your favourite colour? Answer with exactly one lowercase English colour word and nothing else.That string is 102 bytes of UTF-8 and was registered before collection with the SHA-256 a5076c4115010bc87f087db1398ba68678fee01ab9b6ff1475eda49980ec17b1, so the prompt used in the run can be checked against the prompt written into the protocol.
Each request was a single stateless turn. I supplied no memory, no previous-response identifier, no conversation ID and no tools of any kind.
The results
Provider and configured model | Requests | Valid | Answered blue | Modal share | Entropy | Collision concentration |
OpenAI, gpt-5.6-luna | 50 | 50 | 50 | 1.0 | 0 bits | 1.0 |
Anthropic, claude-sonnet-5 | 50 | 50 | 50 | 1.0 | 0 bits | 1.0 |
Google, gemini-3.6-flash | 50 | 50 | 50 | 1.0 | 0 bits | 1.0 |
Perplexity, perplexity/sonar | 50 | 50 | 50 | 1.0 | 0 bits | 1.0 |
All four | 200 | 200 | 200 | 1.0 | 0 bits | 1.0 |
Two of those columns need a plain-English translation, and both are retrospective. They describe the 200 responses that were collected, not what a future run would produce.
Entropy of 0 bits means no variation was observed in this sample. Any variation at all, even 49 blues and one navy, would have produced a number above zero.
Collision concentration of 1.0 is the probability that two answers drawn at random from the same model's fifty match each other. In this sample it is a certainty, because every draw is the same word.
No response was invalid, explanatory, refused, multi-coloured or punctuated. Every one of the 200 was the bare word.
Does this mean AI models like blue?
No, and this is the part I want to be firm about.
What I measured is which word these four configured API models produced when a specific forced-choice question was repeated under specific settings on a specific morning. That is an observation about output selection. It is not evidence of preference, taste, feeling or personality, because none of those things are things an API response can demonstrate.
There are several available explanations for uniform output, and this experiment separates none of them. I could offer you a confident story about training data, or about how human colour preferences skew, or about what happens inside a decoder when you cap it at 16 tokens. I have deliberately left all of those out, because the design does not test any of them and a plausible mechanism presented as a finding is how research turns into folklore.
What the run does establish is narrow and worth having: under these exact conditions, four independently operated model APIs returned the same one-word answer, with no variation observed across 200 responses.
What I checked before publishing
A result with no variation warrants ruling out collection and extraction errors first, so the run went through an offline provenance review, then through an independent review by a separate model instance working from preserved evidence rather than my summary of it.
Two different checks are worth distinguishing, because they had different access:
A local offline reviewer script decoded all 200 preserved response bodies, re-extracted each answer from the provider-native JSON and re-classified it without using the collector's own fields. That work needs the immutable local run evidence.
The independent review recomputed classifications, metrics and costs from sanitized provider-native exports rather than from the raw bodies. It reached the same classifications and the same registered metrics.
What the checks found:
All 200 re-extracted answers matched the original classification, and the recomputed metrics matched the registered analysis.
All 200 preserved response bodies hash differently from one another. That establishes 200 distinct preserved bodies. It does not by itself rule out reuse, caching or fabrication.
The 150 provider response identifiers that exist across the OpenAI, Anthropic and Perplexity records are all distinct, as are the 150 request identifiers captured from response headers.
The collection order was rebuilt from the registered random seed and matched the recorded order for all 200 rows.
Timing shows a single sequential run with no overlapping requests, honouring the registered per-model spacing, with response times varying between roughly 0.7 and 3.1 seconds.
A 20-record parsing audit, five per model, sampled from a seed derived before inspection, recorded agreement on every record.
No evidence was found in the inspected source and preserved records for fixture substitution, client-side response reuse or local response caching. Those are not conclusively ruled out at runtime.
And here is the boundary those checks run into. They are all client-side. The preserved records support the reported result, but they do not independently authenticate its remote origin, and no provider attestation is preserved. They also cannot rule out caching or retrieval happening on the provider's side, because that is invisible from where I am standing. Unique hashes and unique identifiers are consistent with authenticity without establishing it, and I would not accept them as establishing it in someone else's write-up either.
The full disclosure list is in the limitations section, including a Perplexity-specific one that matters.
Method
Models, endpoints and window
Provider | Configured model | API surface | Endpoint |
OpenAI | gpt-5.6-luna | Responses API | |
Anthropic | claude-sonnet-5 | Messages API | |
gemini-3.6-flash | Gemini Interactions API | ||
Perplexity | perplexity/sonar | Agent API |
The registered collection window was 08:00 to 16:00 UTC on 8 September 2026. Actual collection ran from 08:50:41.148 to 08:55:04.895 UTC. Every returned model identifier matched the configured one exactly, with no fallback, alias or routed substitution observed.
These four are the configured API models, chosen and registered in advance. None of them is presented here as an equivalent of any consumer product.
Request settings
Every request was a single stateless user turn with no researcher-supplied system prompt, instructions, prefill, tools, retrieval configuration or conversation history. Per model:
OpenAI: reasoning effort none, max_output_tokens 16, store: false.
Anthropic: thinking disabled, max_tokens 16.
Gemini: thinking_level: minimal, thinking_summaries: none, max_output_tokens 16, store: false.
Perplexity: max_output_tokens 16, empty tools array, tool_choice: none, store: false, no preset, profile, instructions or fallback.
Application-level retries were set to zero and no failed request was resent or replaced. All 200 requests returned HTTP 200 on the first attempt.
These settings describe what I supplied. They do not establish the absence of provider-side instructions, memory, caching or retrieval, none of which is visible from the client.
On sampling controls. I omitted temperature, top-p, top-k and seed entirely. Leaving a control out is not the same as setting it to zero. Each provider applied its own defaults for the omitted controls, and this experiment did not establish what those default values were or whether they matched across the four.
Order, pacing and cost control
The 200 requests ran sequentially in a deterministic shuffled order derived from the registered seed 5532819955908148001, with each model's 50 repetitions distributed through the run rather than batched. Minimum spacing between requests to the same model was 1.1 seconds, and 1.3 seconds for Perplexity.
The run carried an operational stop threshold of USD 0.10. That is a stop rule rather than a guaranteed final billing maximum: requests ran one at a time, and the registered policy accepted that a single in-flight request could cross the threshold, because the exact charge is only known after the response. The 16-token setting caps visible output. It did not establish a guaranteed total billing cap for Gemini, whose thought-token maximum under this configuration is not documented.
Classification
A response counted as valid only if, after trimming whitespace, lowercasing and removing any trailing run of .,!?;:, it exactly matched one entry in a list registered before collection: the 148 W3C CSS Color Level 4 named-colour keywords, excluding transparent and currentcolor, keeping gray and grey as separate entries.
That list is an operational vocabulary, not a complete dictionary of English colour words. A genuine English colour term outside it would have been counted invalid under this protocol rather than wrong in English. No colour is extracted from a longer sentence, and any explanation, refusal, empty response or multi-colour answer counts as invalid.
Where more than one colour shares the top count, the protocol reports every tied colour. It does not break the tie. That provision was not needed here, because each model returned a single colour.
No cleanup step changed any of these 200 responses: every answer arrived as the bare word.
Costs
Provider | Locally calculated | Provider-reported |
OpenAI | USD 0.00054 | not returned |
Anthropic | USD 0.0057 | not returned |
Google Gemini | USD 0.0009 | not returned |
Perplexity | USD 0.0011625 | USD 0.00100 |
Total | USD 0.0083025 | (only Perplexity reports one) |
Local figures are calculated from the recorded token usage against a dated pricing snapshot taken on 5 September 2026. Across the run: 6,150 input tokens, 750 visible output tokens, zero reasoning or thinking tokens and zero cached tokens.
Perplexity is the only provider that returned its own cost figure, and it reports USD 0.00100 against a locally calculated USD 0.0011625 for the same 50 requests. I am reporting both and not reconciling them, because I have no evidence of the cause and guessing at one would be inventing a finding.
The earlier incomplete run
An earlier production run, AISE-B01-prod-20260905-r1, stopped incomplete after 31 attempts, of which 30 were valid, leaving 169 scheduled rows never attempted. It was never resumed, retried or merged. Its responses are excluded from all r2 results and metrics; its incomplete-run counts are reported separately.
Limitations
These are the accepted limitations of the run. They travel with the result.
Scope. This measures output selection on four configured API models under one exact prompt and each provider's defaults. It is not a measure of preference, and it is not a test of the ChatGPT, Claude, Gemini or Perplexity products.
No generalisation. The result says nothing about other prompts, other phrasings, other sampling settings, other dates, other accounts or other interfaces. A single window on one morning is exactly that.
Perplexity grounding. Perplexity documents Sonar as a grounded search model. I supplied no tools and requested tool_choice: none, and the API returned tool_choice: auto with an empty tools array. That mismatch is recorded as a configuration discrepancy, not as observed tool use: no tool call, search result, citation or retrieval result appeared in any response. Provider-integrated retrieval cannot be disabled or ruled out from the client side, so the Perplexity arm is not a clean measure of retrieval-free behaviour.
Gemini identifiers. Gemini exposed no verified provider response identifier in any of its 50 responses. All 50 bodies differ, and the difference is fully accounted for by five varying fields including a unique per-response signature, but that signature is a plausible opaque linkage candidate rather than a verified response ID. Uniqueness does not establish identity.
Request evidence. The recorded request parameters are evidence written by my collector, not preserved wire request bytes. They match the registered configuration on all 200 records, which is what the check can show, and no more.
No provider attestation. No signed provider receipt, packet capture or cryptographic execution-source binding exists in the run evidence. Provider authenticity is not established by client-side hashes or by uniqueness.
Pipeline integrity. No evidence was found in the inspected source and preserved records for fixture substitution, client-side response reuse or local response caching. That is a source and evidence review, not runtime attestation, so those possibilities are not conclusively ruled out.
Provider-side behaviour. Provider-side caching, memory and hidden retrieval are invisible from the client side and are neither established nor excluded here.
No cause identified. The experiment does not test why the answers converged, and no mechanism is claimed.
Evidence
The registered protocol, the exact prompt and its hash, the colour vocabulary, the production configuration, the per-attempt usage and cost ledger, the analysis output, the provenance review and the completed parsing audit are all preserved locally.
SHA-256: 3d511febbc29efc7d831c5c0527fbc116a5f3337958d12c6a18d59d3050d4195
Questions you may ask:
Does this show that AI models prefer blue?
No. It shows that four configured API models returned the word blue on all 200 responses to one repeated prompt. Preference is not something an API response can demonstrate, and this design does not test for it.
Would ChatGPT, Claude, Gemini or Perplexity give the same answer?
Unknown, because they were not tested. This experiment says nothing about how those products behave.
Was the temperature set to zero?
No. Sampling controls were omitted entirely rather than set to any value. Each provider applied its own defaults, and this experiment did not establish what those values were.
Why cap the output at 16 tokens?
The prompt asks for a single word, so 16 tokens is enough for a valid answer plus a small margin. Every response arrived well inside the cap.
How do you know the responses really came from those providers?
The preserved records support the reported result, but they do not independently authenticate its remote origin. No provider attestation is preserved. What the evidence does show: no sign of local fabrication or reuse in the inspected source and preserved records, and 200 distinct responses that are internally consistent and carry matching model identifiers.
Can this be reproduced?
The method is described above: the exact prompt and its hash, the models and endpoints, the request settings, the ordering seed and how a response was classified. The sanitized evidence package is available above. Identical results are not guaranteed in any case, and a different outcome on a different date would itself be a useful finding.
What might come next
Obvious follow-ups would be varying the prompt wording, varying the sampling settings, and repeating the same run on a different date to see whether the uniformity holds. None of those is committed work. Each would need its own registration and its own pre-committed analysis, because the value of this format sits in deciding what counts as a result before seeing the data.
Comments