Does robots.txt Actually Stop AI Crawlers?

Ask five technical SEOs whether robots.txt against AI crawlers actually works and you'll get five confident, contradictory answers. Lately most of them are wrong in one direction or the other. The honest answer is that robots.txt AI crawler blocking works exactly as designed, for exactly the crawlers that were always going to respect it, and it does nothing against the ones that never intended to ask permission in the first place. Both halves of that sentence are backed by real, dated evidence, not vibes, and the failure mode isn't the file. It's treating one text document as a single yes-or-no answer to a question that has two very different populations behind it.

Why robots.txt vs AI crawlers is not a yes/no question

The debate over robots.txt AI crawlers gets framed as a binary because it's convenient to argue that way. One camp points at spoofed user-agents and bypass rates and declares the file dead. The other camp points at server logs showing diligent, repeated compliance and declares the whole conversation overblown. Both camps are reading real data. They're just reading data about different crawlers.

Robots.txt was never a lock. It's a posted sign. It works precisely as well as the discipline of the entity reading it, and AI crawling in 2026 is not one entity. It's a spectrum that runs from Google, Anthropic, and OpenAI's verified, publicly documented crawlers on one end, all the way to unbranded scrapers and crawlers actively disguising themselves on the other. A single robots.txt file gets a different answer from every point on that spectrum, and most of the public argument treats it as if it only had to answer to one.

Read those three numbers side by side and the shape of the real problem comes into focus. It isn't that robots.txt fails. It's that it fails selectively, against a specific slice of traffic, while succeeding loudly and repeatedly against the crawlers a site actually wants visiting in the first place. Sites that only track the failure rate conclude the file is worthless. Sites that only track the compliance rate conclude they're covered. Neither conclusion survives contact with both datasets.

The case against robots.txt: AI crawlers that don't ask permission

Start with the accusation that made the loudest noise this year. Cloudflare, which runs infrastructure for a meaningful share of the web and therefore has visibility few other companies can match, has publicly accused Perplexity of evading Cloudflare's own AI-crawler blocks through stealth tactics. The allegation is specific: rather than declaring itself honestly through its known crawler user-agent, Perplexity's traffic was found spoofing generic browser strings to disguise automated scraping as ordinary human visits, continuing to pull content from sites that had explicitly, deliberately blocked AI crawlers. This wasn't a rumor confined to a niche forum. It was widely reported across tech press, and it came from the company best positioned to see the pattern across thousands of customer properties at once.

“A blocked crawler that wants back in doesn't negotiate with your robots.txt file. It just stops telling you who it is. That's what Cloudflare says happened with Perplexity, and it's the exact scenario a text file has no mechanism to stop.”

Cloudflare's accusation matters because it's a single, high-profile example of a pattern that shows up systemically once you look at the aggregate numbers. A July 2026 report, the AI Crawler Block Index, analyzed robots.txt compliance across more than 10,000 domains and found that 39.5% of declared GPTBot bans were not actually honored by the traffic hitting those sites. Roughly two out of every five sites that told GPTBot to stay out, in writing, in the one file crawlers are supposed to check first, got crawled anyway. That's not a rounding error or a handful of misconfigured edge cases. At more than ten thousand domains sampled, a bypass rate over a third is a structural feature of how AI crawling behaves against declared blocks, not an anomaly.

The clearest confirmation that this problem is real, rather than a talking point, is who acted on it. Cloudflare didn't just publicize the Perplexity allegation and move on. It announced a new policy under its BotBase model, taking effect September 15, 2026, under which Verified Bots that ignore a site's declared robots.txt preferences, or that reproduce a site's content in full rather than summarizing or citing it, risk losing their Verified status entirely. That's a meaningful admission from the one company with the broadest network-level view of this traffic: robots.txt compliance can't be assumed. It has to be actively policed, with real consequences attached, or the declared-blocks system doesn't hold. If passive trust in a text file were sufficient, Cloudflare wouldn't need a verification program built around punishing the crawlers that break it.

The case for robots.txt: the AI crawlers that read every line

None of that means robots.txt is a dead letter for the crawlers that matter most, and the counter-evidence is just as concrete. Dan Petrovic, founder of DEJAN, a technical GEO research firm, published a 30-day server log analysis of his own site, dejan.ai, on August 15, 2026. He didn't survey opinions or extrapolate from a sample of headlines. He pulled his own logs and counted actual requests over an actual month.

What he found runs directly against the cynical read that big AI labs treat robots.txt as a formality. Over that 30-day window, Google's, Anthropic's, and OpenAI's verified crawlers fetched dejan.ai's robots.txt file a combined 1,150 times. In the same period, the same crawlers fetched the site's llms.txt file, the newer machine-readability convention that's been pitched as the future of AI-crawler guidance, only 4 times. That's not a close comparison. It's a nearly 300-to-1 ratio in favor of the older, blunter, unglamorous file over the format that's supposed to be replacing it, and it comes straight from server-side request logs, not survey data or self-reported behavior.

The takeaway from Petrovic's data isn't just that robots.txt gets checked. It's that it gets checked obsessively, far more than the file most GEO advice in 2026 tells sites to prioritize. That squares with what we found when we looked at whether llms.txt actually works and separately at the finding that most published llms.txt files go unread 97% of the time: the newer, purpose-built convention gets ignored by the crawlers that matter most, while the older, general-purpose file keeps getting hit, repeatedly, by name-brand operators that have every incentive to follow it. A verified, honestly-identifying crawler checking robots.txt over a thousand times in a month isn't a crawler treating the file as optional.

Our verdict: robots.txt works, for an audience that keeps shrinking

Both sides of this argument are citing real data, and neither side is describing the whole picture on its own. The verified, honestly-identifying crawlers from Google, Anthropic, and OpenAI, the ones a site actually wants making legitimate, attributable, citation-worthy visits, respect robots.txt and check it constantly. Petrovic's own logs prove that beyond a reasonable doubt. Robots.txt is not some legacy artifact the major labs quietly stopped honoring the moment nobody was watching; the request volume says the opposite.

But a meaningful minority of traffic isn't playing that game at all. Some of it is outright bad-faith identity spoofing, which is exactly what Cloudflare alleges Perplexity did to keep scraping sites that had explicitly blocked it. Some of it is unverified, unbranded scraping infrastructure riding the general AI wave with no reputation to protect and no verification program to lose, which is the likeliest explanation behind the AI Crawler Block Index's 39.5% bypass figure across more than 10,000 domains. Neither of those actors can be stopped by a user-agent string in a text file, because a user-agent string is just a claim, and claims can be faked as easily as they can be honored.

Cloudflare's own shift is the clearest institutional proof that this two-part conclusion is correct. A company running infrastructure at that scale didn't need to build an active-verification program if passive robots.txt trust were already working across the board. It built one specifically because the compliant majority was already handled and the noncompliant minority wasn't, and September 15, 2026 is when that policy starts actually revoking Verified status from crawlers caught ignoring declared preferences or reproducing content wholesale. That's the exact same conclusion this article is making, reached independently, at infrastructure scale, by the company with the best visibility into the problem.

The two-layer fix: robots.txt plus real AI crawler verification

The practical move isn't to abandon robots.txt, and it isn't to treat it as a finished job either. It's to run two layers at once, because they protect against two different populations of crawler.

The first layer is cheap, and Petrovic's log data is the argument for taking it seriously rather than treating it as a box-checking exercise. If Google, Anthropic, and OpenAI's crawlers are checking your robots.txt over a thousand times a month, a stale or generic file is actively costing you control over crawlers that would otherwise follow specific, granular instructions. This is the same instinct behind building a genuinely agent-readable site: the crawlers worth optimizing for are the ones reading carefully, so give them something precise to read.

The second layer is where most sites are exposed, because it's the layer a text file was never built to provide. Bad actors and unverified scrapers can put any string they want in a User-Agent header, and neither the Cloudflare-alleged Perplexity spoofing nor the Block Index's 39.5% bypass figure could have been stopped by a more strongly worded Disallow line. They require actually confirming that the traffic claiming to be GPTBot, ClaudeBot, or Google-Extended is coming from the IP ranges those operators actually publish, or resolves correctly through reverse DNS to their documented domains. That's infrastructure work, not a robots.txt edit, and it's the part of this problem most sites still haven't touched.

If you're running a technical stack where API documentation, changelogs, and product content are exactly the pages AI crawlers hit hardest, this gap matters more, not less. That's especially true across devtools companies, where crawler traffic against docs and reference pages is often the single largest source of non-human load, and where the line between a legitimate verified crawler and a scraper pretending to be one determines whether that traffic is worth the server cost. Knowing which crawlers are actually respecting your declared preferences, and which are just claiming to, isn't something you can determine by reading a robots.txt file in isolation. It takes checking it against real request logs the way Petrovic did, and against real IP and DNS data the way an edge-verification setup does.

That's the gap most sites are sitting in right now: a robots.txt file that's either untested against their own logs or untouched since before AI crawlers existed, and no verification layer behind it at all. Running a technical access audit against both is the fastest way to find out which side of this debate your site is actually on, rather than guessing based on a headline about Perplexity or a single blog post about Google's crawl volume. The honest answer is almost never all-or-nothing. It's a specific list of which crawlers are honoring your rules, which ones are quietly ignoring them, and which ones are lying about who they are.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.