The robots.txt Carve-Out That's Eating Your Content Controls
2026-08-15 — brand building small business India
I watched a client block ChatGPT-User in their robots.txt, then watch it fetch their pages anyway. They thought the file was law. It isn't.
The loophole is simple, more or less. OpenAI's documentation says ChatGPT-User visits a page when a ChatGPT user asks a question, and because those actions are initiated by a user, robots.txt rules may not apply. Not "do not apply." May not. The hedge matters — but so does what happens in practice.
User-initiated. That's the escape hatch.
The data is worse than the policy. TollBit's latest State of the Bots report shows ChatGPT's page-fetching bot is disallowed by more sites than any other AI bot of its kind, yet it also reached disallowed pages on more sites than any other bot. In Europe, ChatGPT-User, Bytespider, and Youbot each accessed disallowed pages on nearly half of the sites that explicitly blocked them. ChatGPT-User hit the most. If you blocked it, it still came through — because someone in Beijing or Brooklyn asked it to.
Here's where most consultants get this wrong, including us sometimes.
People assume robots.txt controls ChatGPT's fetch behaviour. It doesn't. OpenAI runs a three-bot system: GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for user-initiated retrieval. That last one — the one fetching in real time because a human typed a question — is the one OpenAI says maybe doesn't listen to you.
OAI-SearchBot, the crawler that decides if you show up in ChatGPT search, is different. You can block that. But ChatGPT-User? ChatGPT-User operates outside those rules, and it's becoming more capable — not less.
The argument is defensible if you squint. When a person asks ChatGPT a question, the bot becomes an extension of that person's browser, something like a proxy. Proxies don't usually check robots.txt — they're the user's tool, not a crawler. But then, no crawler actually tells you it was "triggered by a user" either. Every bot could claim that. OpenAI's rationale is that because these actions are user-initiated, the agent is not considered an autonomous indexing or scraping system, thus bypassing the rules intended for general-purpose crawlers.
Actually, that's not quite right. Let me back up.
The real problem isn't the philosophical argument. It's that the web ran on an honour system for thirty years, and the honour system is dead. Perplexity-User generally ignores the robots.txt file for the same reason, but Anthropic has a different view and states that all three of its bots respect it. So now you have three companies, same structure, three different interpretations. One respects your file. One doesn't. One says maybe.
The fix is supposed to come from the network layer. Cloudflare is launching the ability to manage AI traffic based on three major use cases: Search, Agent, and Training crawlers. Starting on September 15, 2026, Cloudflare's default settings will block "mixed-use" crawlers from any pages that host ads.
That's teeth. Not a request. A block.
But Cloudflare only protects customers who use Cloudflare. Most of the web doesn't. And the carve-out for user-initiated requests survives. Everyone builds agents that fetch pages this way now — Perplexity, Anthropic, Google. If the user-initiated loophole holds, it holds for all of them. The network layer can't touch it without breaking the fundamental idea that a user controls their own browser.
The old handshake isn't coming back.