Writing & Research

Two things live here. Posts about organic growth and AI search, written when I have something worth adding. And AI Search Experiments, the research series I am currently running.

Most of what is here came out of work I was doing anyway.

AI Search Experiments

AI Search Experiments is a research series I am running on AI search and model behaviour. Small studies, one question at a time.

The approach is method first. Each study sets out what it asked, how it was run, and what it cannot tell you, in the same place as the result rather than tucked away from it. If something turns out to be wrong after publication, the correction goes on the study itself.

AISE-001: citation repeatability

The first study looks at citation repeatability across particular AI model and retrieval stacks, accessed through their APIs. It is a question about the systems underneath the products rather than about consumer chat apps. Across 20 question-and-stack cells, the median cited-domain overlap between repeats of the same question was 42.4%, and the write-up spends as long on what that does not prove as on the number.

Ask the same question five times, and the cited sources move

Heatmap of median cited-domain overlap for four AI provider API stacks across five questions. OpenAI: 67, 100, 83, 33 and 20 per cent. Anthropic: 44, 37, 44, 47 and 25 per cent. Gemini: 23, 38, 38, 31 and 45 per cent. Perplexity Agent API: 62, 57, 57, 40 and 40 per cent. Columns are project management, email marketing, language learning, carry-on luggage and photo books. Overall median across the 20 cells is 42 per cent as displayed and 42.4 per cent to one decimal place.

Ask the same question five times, and the cited sources move

Five registered questions, all built from one locked template, run through four provider APIs with web search enabled, five times each. That is 100 answers in 20 question-and-stack cells. Inside each cell I measured how much the cited domains overlapped between two repeats of the same question, and the median of those 20 within-cell medians was 42.4%. The full method, the table and the limits are all here.

1 day ago9 min read