ChatGPT's Citations Don't Match What Google or the API Can Find

2026-10-19 — stakeholder management consulting

Here's what happened when researchers tried to replicate ChatGPT's citations using standard tools. They couldn't. Not even close.

They captured what the web app was actually citing in real time by watching the data flow during each session. Then they tried to reconstruct those same citations using two proxies: Google's organic search results and OpenAI's own Responses API with web search enabled.

The gap was massive.

That's the key asymmetry here. If you're tracking at the domain level, you look better than you actually are. At the page level—the thing that actually matters for accuracy—you're blind to three-quarters of what ChatGPT used.

The test was rigorous enough. Each question ran 20 times through GPT-5.3 Instant and GPT-5.5 Instant. Google contributed about 43 results per question on average. The API search had web search enabled. Everything was queried within three days of the ChatGPT web app runs to keep variables tight.

Inconsistency was the real story, though. Across 20 repeat runs, 40.8% to 44.5% of retrieved URLs appeared only once. Run the same prompt twice, get different results. Run it four times, get even more fragmentation. This isn't noise. It's systematic drift.

In the study, GPT-5.3 Instant and GPT-5.5 Instant never queried using the exact wording of the original question, while the older GPT-5.2 Instant did so in only 5.5% of runs. That means if you're trying to replicate a ChatGPT search by just copying the user's original prompt and throwing it at Google or the API, you've already lost the thread.

There's another wrinkle. The model translates silently and searches in a different language. A tracking tool that doesn't know this is tracking the wrong thing.

Most consultants and in-house teams get this wrong, including us sometimes. They assume that if you can find a citation domain in Google results or reconstruct it via API, you've understood the sourcing chain. You haven't.

The researchers suggest monitoring both at the page level and the site level, running the same prompt multiple times to catch variability, and staying aware that ChatGPT reformulates queries. They also note that commercial tracking tools—the ones you might actually use—weren't tested in the study. So the gap between what these results show and what your existing tracking stack can actually do is probably wider.

The arXiv preprint arrived on October 11. It's not peer-reviewed. But if you've been assuming that ChatGPT's citations are verifiable through standard proxies, the assumption just cracked open.

Source: "Google search results and OpenAI's own Responses API do a poor job of reproducing the exact pages ChatGPT cites." — Relevant Audience