Every AI-crawler-economics conversation eventually lands on a version of the same number: crawlers request a lot, they refer back very little. Cloudflare's own July 2026 data, published as it explained why it moved from charging per crawl to compensating publishers per citation, put fresh figures on that gap: Anthropic's crawler requested roughly 38,000 pages for every one referral it sent back, and OpenAI's sat around 1,091 to one. What got less attention in the coverage is the second finding buried in the same research, run jointly with ETH Zurich: more than 90% of the pages these bots crawl are unique content, not repeat hits on the same handful of popular pages. That detail matters more than the ratio itself, because it's the reason the standard fix for expensive crawler traffic, caching, doesn't actually work here.
What AI crawlers actually cost: the numbers
Cloudflare sits in front of a meaningful share of the web, which is exactly why its own crawl-to-referral figures carry weight independent of the third-party audits we've covered before. In this dataset, Anthropic's crawler requested about 38,000 pages for every one referral it produced, and OpenAI's sat closer to 1,091 requests per referral. Those numbers won't match every other analysis of the same crawlers, including TechnologyChecker.io's independent robots.txt study, which measured different ratios over a different window using different methodology. That's expected, not a red flag: different vantage points, sampling windows, and definitions of what counts as a 'crawl' or a 'referral' will produce different absolute numbers. What holds across every version of this analysis we've seen this year is the shape of the finding: AI crawlers request enormously more than they send back, by one to several orders of magnitude more than a conventional search crawler like Googlebot.
Cloudflare published this alongside its move from Pay Per Crawl, a straightforward per-request charge, to a model that compensates publishers based on whether their content actually gets cited in an answer. We covered that shift in detail when it happened. This piece is about a different, more technical question sitting underneath the monetization debate: why does this traffic cost so much to serve in the first place, and is there an engineering fix short of blocking or charging for it.
Why caching doesn't save you here
Every site with meaningful traffic relies on caching to keep origin-server costs manageable. The logic is simple: a small number of pages get requested constantly, a CDN or edge cache serves those from memory instead of hitting your database and application servers every time, and your infrastructure bill stays proportional to unique content, not total requests. It's why a spike in traffic to your homepage or a viral blog post barely moves your server costs. The cache absorbs it.
The joint Cloudflare and ETH Zurich research found that AI crawlers break this assumption almost completely. Rather than repeatedly requesting a small set of popular URLs, these bots spread their requests across a long tail: over 90% of what they crawl is content the cache hasn't already got warmed, because it isn't the small set of pages getting hit over and over. A crawler systematically working through your entire product catalog, documentation set, or archive isn't behaving like a traffic spike on a hot page. It's behaving like a full site re-index, every time, and a cache built to absorb repeat requests on popular content does very little for a traffic pattern built entirely out of one-time requests to pages nobody else is asking for right now.
| TRAFFIC PATTERN | TYPICAL BOT (SEARCH CRAWLER, HUMAN SPIKE) | AI TRAINING/RETRIEVAL CRAWLER |
|---|---|---|
| Request distribution | Concentrated on popular, frequently-updated pages | Spread across long-tail, rarely-requested pages |
| Cache hit rate | High, once warmed | Low, per Cloudflare/ETH Zurich's 90%+ unique-content finding |
| Cost driver | Origin load during traffic spikes | Origin load on effectively every request |
“A cache built to absorb repeat requests does very little against a crawler whose entire traffic pattern is one-time requests to pages nobody else is asking for.”
This is a different problem than robots.txt
It's worth separating this from the access-control conversation we've had before around AI crawlers accidentally blocked by stale robots.txt rules or the rising 403 rejection rate as more sites deliberately shut bots out. Robots.txt and 403 responses are a binary gate: a crawler either gets in or it doesn't. The caching problem exists on the other side of that gate entirely, for every crawler you've deliberately chosen to allow. Even a site with a clean, intentional robots.txt policy that welcomes GPTBot and ClaudeBot for good GEO reasons still pays the infrastructure cost this data describes, because welcoming the crawler doesn't change the shape of its request pattern. You can solve the access question and still have an expensive-to-serve problem sitting on the other side of it.
This also connects to why Cloudflare's own economics argument for AI crawlers is stronger than the equivalent argument for traditional search crawlers. Googlebot's crawl-to-referral ratio, by comparison, sits close to 5:1 in most third-party analyses, and its requests concentrate on pages that also get real human traffic, which is exactly the pattern caching is built for. That's a meaningful part of why nobody built a metered-access business around Googlebot the way Cloudflare is now building one around AI training crawlers: Googlebot's traffic was never expensive to serve in the way this one is, both because the ratio is smaller and because the request pattern actually plays nicely with standard infrastructure.
Where AI crawlers hit hardest: large, deep sites
The caching problem scales with how much long-tail content a site actually has. A five-page marketing site has almost nothing for a crawler to spread across, so this whole dynamic barely registers. A large enterprise property, a documentation set with thousands of versioned pages, an ecommerce catalog with deep category and product pages, or a media archive going back a decade, is a completely different story. These are exactly the sites where a crawler working systematically through the long tail generates the most origin load relative to what caching can absorb, because the long tail is most of what the site is. If you're running custom software or a large site build, the AI-crawler cost conversation belongs in the same architecture discussion as any other scaling concern, not bolted on afterward as a robots.txt tweak.
It also changes how you think about content pruning. Sites carrying years of thin, outdated, or near-duplicate pages have historically tolerated that bloat because human traffic and even Googlebot mostly ignore it. AI crawlers don't extend the same courtesy. A large, low-value archive is exactly the kind of long-tail surface these bots are shown to spend disproportionate crawl budget on, which means the cost case for consolidating or removing genuinely low-value pages just got a second justification beyond the usual content-quality argument: every stale page left live is also a page an expensive crawler is going to request, uncached, indefinitely.
What a tiered access policy actually looks like
Given both a bad ratio and a caching problem, the reasonable response for most sites isn't a blanket allow or a blanket block. It's differentiated treatment by crawler and by content type.
Do this next
Pull your server logs this week and separate AI crawler traffic by bot, not as one lump category. If you're on Cloudflare or a similar CDN, check whether it exposes crawl-to-referral data per bot directly, since that removes the guesswork. Then decide, deliberately, which crawlers earn full access to your most valuable pages and which ones get throttled on the long tail. That decision is more useful than either extreme, blocking everything out of cost anxiety or leaving everything open because GEO advice says access matters. Access matters. So does not paying to re-index your entire archive for a crawler that sends you one visit for every 38,000 pages it takes.
None of this changes the case for staying reachable to the crawlers that actually convert into citations. It just means treating crawler access as an infrastructure decision with real cost tradeoffs, the same way you'd treat any other traffic pattern that doesn't behave the way your caching layer expects, instead of a single generative engine optimization checkbox you either check or don't. The teams that get this right in the next year won't be the ones with the strictest robots.txt or the most generous one. They'll be the ones who actually measured which crawler earns its keep, on which pages, and built infrastructure decisions around that answer instead of a blanket policy borrowed from someone else's site.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.