AI crawler blocking doubled this year. The robots.txt data on who's winning.

If you're deciding whether to block AI crawlers, the honest starting point is that a lot of the internet has already decided, and decided harder than it did last year. A robots.txt and network-traffic analysis from TechnologyChecker.io found that 403 Forbidden responses to AI bots more than doubled year-over-year, from 3.63% to 8.56% of AI-bot requests. That's not a rounding shift. That's the web's default posture toward AI crawlers moving from mostly-open to meaningfully-defended in twelve months, and the crawl-to-referral data behind it explains why site owners pulled the trigger.

The study, published by TechnologyChecker.io's CTO David Thomson and updated in July with year-over-year figures, is built on Cloudflare Radar network data spanning 330 cities in 125-plus countries, cross-referenced against a robots.txt sample of over 4,000 files parsed at the start of each quarter. That's a meaningfully larger and more current dataset than most of the robots.txt surveys that circulated in 2025, and it's the first one we've seen with a clean quarter-over-quarter blocking-rate comparison. Read the full breakdown at TechnologyChecker.io's robots.txt analysis if you want the raw methodology. Read on if you want what it means for the block-or-allow decision sitting on your desk right now.

The number: 403s against AI crawlers more than doubled

Start with the headline figure, because it's the one that should reset your assumptions about how contested this space has become. In Q2 2025, 3.63% of requests from AI bots hit a 403 Forbidden response. By Q2 2026, that figure was 8.56%. Same crawlers, largely the same set of operators, more than double the rejection rate in a single year.

Put that in context against the rest of the traffic picture the same report surfaced: bots now make up roughly 30.6% of all web traffic as of Q1 2026, and AI crawlers specifically account for a growing, unevenly distributed slice of that. Sites aren't blocking AI crawlers because AI crawling is rare. They're blocking it because it's become a large, constant, and largely one-directional load on infrastructure, and enough site owners have now run the numbers to act on it.

That last point matters. A move from 3.63% to 8.56% isn't a handful of publishers making noise on a blog. It's a broad, distributed shift across a 4,000-plus-site sample, which means it's showing up in ordinary robots.txt files, not just the ones belonging to companies with a stated AI-licensing dispute. Whatever caused this, it's mainstream now, not a fringe reaction.

Why more sites now block AI crawlers

The blocking-rate jump makes a lot more sense once you look at what each AI crawler actually gives a site back for the privilege of scraping it. TechnologyChecker.io's report includes crawl-to-refer ratios by operator: how many pages a crawler fetches for every one visit it eventually sends back via a citation or referral click. The gap between operators is enormous, and it's the real driver behind the blocking decision, not general AI hostility.

Read that table as a spectrum, not a blacklist. Googlebot and DuckDuckGo sit close to parity because search has always been a value exchange: crawl the page, send traffic back when it ranks. PerplexityBot and Microsoft's crawlers sit further out, but they're still functioning as answer engines that cite and link, which is the entire premise of generative engine optimization working as a discipline at all. GPTBot and ClaudeBot sit in a different category entirely. At 870:1 and 5,143:1, these crawlers are pulling enormous volumes of content relative to what they ever send back, which is exactly the pattern that shows up as unexplained bandwidth and server load with no matching referral-traffic line in analytics.

Meta-ExternalAgent is the outlier worth naming specifically, because it isn't a lopsided ratio in the ClaudeBot sense, it's a limit case. TechnologyChecker.io's sample recorded no referrals from it at all, and just as tellingly, found it absent from every explicit ALLOW rule the researchers checked, meaning even the small share of sites that write crawler-specific robots.txt logic aren't choosing to invite this one in. When a crawler shows up nowhere in the traffic that comes back and nowhere in the rules meant to welcome it, that's not really a judgment call. That's the one line item on this list where blocking outright, rather than metering or monitoring, is the straightforward answer.

The report's sector data backs this up from a different angle. AI crawler traffic isn't distributed evenly across the web; it concentrates hard in specific verticals. Retail alone absorbs roughly 28.7% of AI crawler traffic by industry, more than double the next-largest category, computer software, at around 13.4%. Product catalogs, pricing pages, and inventory listings are exactly the kind of high-page-count, frequently-updated content that training-oriented crawlers love to hit repeatedly, and exactly the kind of content that's expensive to serve at scale and rarely drives a return visit through an AI answer citation. If you run an e-commerce site and you've noticed crawler load spiking disproportionately, the sector data says you're not imagining it.

This is also where the report's purpose breakdown is worth sitting with: across the crawl traffic it measured, roughly half was flagged as training-purpose crawling, with a further chunk mixed-purpose, and only a small single-digit share tied to live user-triggered fetches. Most of what's hitting your server isn't answering a user's question in real time. It's stocking a model for later, with no guarantee your site gets named when the answer comes out.

That purpose split is the piece most block-or-allow debates skip past. A crawl tagged "training" isn't going to turn into a citation next week or next quarter; it's inventory for a model that may or may not ever surface your brand by name, on a timeline nobody discloses. A crawl tagged "user action," the roughly single-digit slice, is a live fetch triggered by someone typing a question right now, the closest thing in this dataset to a real-time search click. Treating those two categories as one undifferentiated "AI traffic" bucket, which is what most default robots.txt setups still do, is exactly how a site ends up either blind to a rising cost center or blind to a rising citation channel, often both at once.

Why you shouldn't block AI crawlers wholesale

This is the part where a lot of site owners overcorrect. Faced with a 5,143:1 ratio on one crawler and rising 403 rates across the industry, the reflex is to write a robots.txt rule that disallows every user-agent with the word "bot" in it and call it handled. That's the wrong move, and it's the same wrong move we flagged in our look at sites blocking AI crawlers by accident: broad disallow rules and stale robots.txt files routinely catch crawlers a site actually wants, alongside the ones it doesn't.

The inverse mistake is just as common and gets talked about less: leaving robots.txt wide open to every AI crawler because blocking feels aggressive or because nobody's gotten around to auditing it. That default costs you real infrastructure spend against crawlers that, per this data, are never going to send meaningful traffic back regardless of how much of your site they index. A GPTBot or ClaudeBot crawl doesn't compound into a citation-driven visit the way a Perplexity or Bing crawl increasingly does; it's a one-way transfer, and treating it the same as a search crawler because both show up as "AI" in the logs is a policy failure, not a neutral choice.

There's a middle option that's been live for over a year and gets undersold: metering instead of blocking outright. We've written before about Cloudflare's Pay Per Crawl program as one live example of what a metered, tiered response looks like in practice, and the skepticism around its per-crawl revenue is fair; for most sites, the dollar figure is small. But the revenue was never the useful part. The useful part is that metering forces a decision per crawler instead of a single global switch, which is precisely the granularity this crawl-to-refer data says you need. A 5:1 or 153:1 crawler is worth leaving open. A 5,143:1 crawler is worth metering, negotiating with directly, or blocking, and those are three different answers depending on what your site can absorb, not one answer applied to everything wearing a bot user-agent.

The tiered policy the data actually supports

Put the blocking-rate trend and the crawl-to-refer table together and the conclusion isn't "block AI crawlers" or "allow AI crawlers." It's "differentiate," and the differentiation criteria the data hands you is referral value against crawl cost, crawler by crawler, not category by category.

None of this requires guessing. Pull your own server logs, match user-agents against the ratios in the table above, and you'll know within an hour which crawlers on your site look like Googlebot's exchange and which look like ClaudeBot's. That's the same log-separation work we walk through with clients before we touch a single robots.txt rule, and it's the same discipline behind the technical foundation in our Arnica case study, where getting the crawler-access layer right was a precondition for the GEO results, not an afterthought bolted on later.

Do this next: export 30 days of server logs, isolate requests by AI crawler user-agent, and run each one against the ratio table above. Anything performing like Googlebot or Perplexity stays open. Anything performing like GPTBot or ClaudeBot gets a metering, negotiation, or section-restriction decision this quarter, not a blanket rule you inherited and never revisited.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.