What Google Actually Wants From Your 503s (And Your Robots.txt)

2026-10-25 — customer acquisition strategy India

PHP had emergency security patches last month. Sites went offline. Google put out a reminder: use 503s during the downtime, not 404s or error pages. Simple enough on the surface.

The real lesson buried in that guidance is less simple.

A 503 tells Google your outage is temporary and keeps indexed pages in the index while you work. That's the headline. But here's what most people get wrong: the 503 has to come from somewhere else. If PHP is down, PHP can't serve the 503. It needs a layer that stays running—a CDN routing rule, a temporary server you point DNS at, something external. John Mueller made that point on Bluesky: it doesn't have to be PHP itself.

The catch is timing. Google's crawl-rate documentation says don't serve 503s beyond one or two days. Let it run longer and Google may start dropping URLs from the index entirely. You'd think that would make the message simpler. Instead it adds a hard deadline to an already nerve-wracking patch window.

One more thing—actually, that's not quite right. Two more things. First, don't return a 503 for robots.txt requests. If you do, you block all crawling. Second, avoid serving a maintenance page with a 200 OK status. That tells Google the page exists and the maintenance message is its actual content. That gets indexed as your real page.

The 503 guidance was about crawling, though. About what Google can and can't access while you patch. That's different from what Google does with your content later.

Gary Illyes from Google clarified that distinction in a Search Central video last month. Robots.txt controls crawling access, not usage. The standard robots.txt directives—Disallow, Allow, User-agent—they only dictate what crawlers are allowed to visit. They don't control how systems use the content after crawling it. Illyes called it an access preference protocol, nothing more.

That matters because AI training is different from search inclusion. A site might want to block a specific AI crawler from the path entirely, or it might want to allow crawling but restrict how the content gets used downstream. Robots.txt can do the first through a Disallow rule targeting that crawler's user-agent token. It can't do the second, at least not yet.

Google-Extended is Google's token for managing whether crawled content gets used for Gemini training. Using it doesn't affect whether you appear in Google Search. It's not a separate crawler—it's a policy flag on top of Google's regular crawling. A standard way to express that training preference separately from crawl access exists only as a draft proposal for now. Illyes co-wrote it for the Internet Engineering Task Force. It hasn't been approved.

The URL complexity is different but touches the same nerve about precision. Raw brackets in URLs and their percent-encoded forms—%5B and %5D—they create different URLs as far as Google is concerned. Mueller said he would have assumed they were equivalent and probably gave wrong answers in the past. RFC 3986 reserves those characters, so the spec says you should encode them. When Google finds the same content served by both versions, it crawls each one, identifies them as duplicates, and picks one to keep.

That matters when links and canonical tags mix formats. Google has to guess which version you prefer. Mueller's answer: pick one and use it consistently. Links, canonicals, sitemaps, structured data—same format across all of them. If your site already uses raw brackets throughout, it'll probably keep working. He wouldn't build a new site that way.

These aren't just technical details. They're the difference between a site that survives an emergency patch and one that drops from the index. Between content that trains AI models and content that doesn't. Between a canonical tag that actually works and one that creates invisible duplicate burden.

Source: "Serving a 503 from outside PHP and keep robots.txt reachable." — Search Engine Journal