You know the feeling. The infrastructure dashboard shows a real, sustained jump. Nobody shipped a viral feature. Human traffic, the numbers you actually report on, looks basically flat. And the bill at the end of the month is not flat at all. Somewhere between the analytics dashboard and the billing page, a whole category of visitor is going completely unaccounted for.
The bill nobody budgeted for
That gap between "traffic looks normal" and "the infrastructure bill doesn't" is, for a growing number of site owners, explained by a visitor category that never shows up in a standard analytics dashboard: AI crawlers. TollBit's network data put AI bots behind 1 in every 31 web visits by the end of 2025, and separate tracking shows AI and LLM indexing bots quadrupling their share of overall traffic in eight months, from 2.6% to 10.1%. OpenAI's GPTBot alone grew 305% in that window. None of those visits show up as a session in Google Analytics the way a human visitor does. All of them show up in your server logs, your CDN bill, and your database load.
David Belson, who covers this from Cloudflare's infrastructure side, put the underlying dynamic bluntly in a recent piece for Search Engine Journal: a lot of this traffic comes from bots that were built fast and carelessly, in his words, "the person who didn't know what the hell they were doing yesterday, but vibe coded a bot today and let it loose." That's not a description of OpenAI's or Anthropic's flagship crawlers, which are at least identifiable and somewhat well-behaved by comparison. It's a description of the long tail underneath them: poorly-built automation hitting expensive endpoints repeatedly, bypassing caching, and forcing requests back to origin servers in a way a well-designed crawler wouldn't.
Where the load actually comes from
Roughly 80% of AI crawling activity is estimated to be for model training rather than live query retrieval, which is the detail that should reframe how a site owner thinks about the cost. A live-retrieval crawler, the kind that fetches your page because a user just asked an AI engine a question, is at least transactional: it costs you a fetch, and in exchange there's a chance of a citation and referral traffic. A training crawler costs you the identical fetch and returns nothing, ever, on that specific visit. It's not gathering information to answer anyone's question right now. It's harvesting content to bake into a future model version, with no citation, no referral, and no way for you to know it happened unless you're actually reading your own server logs.
Think about what that actually means for a site with a meaningful catalog or content archive. Every page a training crawler touches costs real compute, whatever it takes to serve that page, whether from cache or, worse, from a full database query if the crawler happens to hit a page or parameter combination that isn't cached. Multiply that by a crawl frequency measured in the hundreds of millions of fetches industry-wide, and the aggregate cost stops being a rounding error on anyone's infrastructure line and starts being a real, if diffuse, tax on every site with enough content to be worth crawling in the first place. Nobody sent an invoice. Nobody asked permission. The cost still landed.
The crawl-to-referral economics Something Inc. has already tracked make the imbalance concrete: ClaudeBot's ratio sat around 1,917 fetches for every one referral visit as of early August, while GPTBot, after some improvement, still ran roughly 251:1. Those aren't rounding errors. They're the actual exchange rate a site is accepting every time it leaves a crawler unrestricted, and the rate is set entirely by the crawler operator, not negotiated with the site absorbing the cost.
| CRAWLER | APPROX. CRAWL-TO-REFERRAL RATIO | WHAT THAT MEANS PER VISIT |
|---|---|---|
| ClaudeBot | ~1,917:1 | Nearly 2,000 fetches absorbed for every one referral visit sent back |
| GPTBot | ~251:1 | Better than ClaudeBot, still a steep cost-to-return ratio |
| Googlebot | ~5:1 | The baseline comparison: a crawler with an established, direct traffic exchange |
Not all crawlers deserve the same policy
The instinct once a site owner sees a bill spike traced to bot traffic is to block everything AI-related and move on. That instinct is understandable and, per research Something Inc. has already covered, often the wrong trade: blocking indiscriminately also blocks the live-retrieval crawlers that can actually send citation-driven traffic back, and a blanket block doesn't distinguish between a well-behaved crawler respecting your rate limits and the "vibe coded" one hammering your checkout flow every few seconds.
The robots.txt data Something Inc. has tracked already shows this shift happening at scale: 403 Forbidden responses to AI crawlers rose from 3.63% to 8.56% year over year, meaning more site owners are reaching for a block. What that data can't tell you is how many of those blocks were a deliberate, tiered decision versus a panic response to a bandwidth spike that a more targeted fix would have solved just as well, at a fraction of the citation-opportunity cost.
There's also a category error worth naming directly: treating "AI crawler" as one bucket at all. A user-agent string claiming to be GPTBot, ClaudeBot, or PerplexityBot is at least a named, somewhat accountable actor with a published purpose and, in most cases, a documented way to verify the request actually came from where it claims. The unnamed, unbranded scraper traffic Belson describes, hastily built, unrate-limited, indifferent to robots.txt entirely, is a completely different risk category, and it's often the larger share of the actual cost even though it gets far less attention than the recognizable brand names driving the GEO conversation. A policy built only around the well-known crawlers misses the traffic doing the most damage to an infrastructure bill.
A policy that doesn't require blocking everything
A workable version of that policy has three parts, and none of them require a full block. Start by identifying which pages are actually expensive to serve, cart and checkout flows, search and filter endpoints, anything that triggers a database query or a server-side render rather than serving from cache, and restrict crawler access to those specifically, rather than the site as a whole. A crawler doesn't need to hit your live search-results endpoint to represent your product catalog accurately; a static, well-structured category page does that job at a fraction of the compute cost, for both a human visitor and a bot.
Second, separate training crawlers from live-answer crawlers in your access policy, since they warrant genuinely different treatment. A live-retrieval crawler fetching a page in response to an active user query is the one with a plausible near-term payoff, a citation, a referral click, brand exposure inside an actual answer. A pure training crawler has none of that; permitting it is a bet on long-term model representation, not a transaction with any visible short-term return, and it's reasonable to price that bet differently in your access decisions than you'd price a crawler with an immediate, measurable upside. Some teams will decide that long-term representation bet is still worth the cost. That's a legitimate call, but it should be a deliberate one, made with the crawl-to-referral numbers in front of you, not a default nobody actually chose.
Third, monitor for the specific pattern Belson flagged: repeated requests, tight loops, and origin-server hits that bypass your CDN cache entirely. That pattern is the signature of a poorly-built bot, not a well-run one, and it's worth restricting on behavior rather than by user-agent string alone, since a user-agent can be spoofed or simply left unset by whoever built the thing in an afternoon. Rate-limiting and anomaly detection at the infrastructure layer catch that pattern regardless of what the crawler claims to be, which matters most precisely because the worst offenders are the ones least likely to identify themselves honestly in the first place.
None of this is a one-time fix. Crawler behavior, volume, and the ratio of good actors to careless ones keeps shifting quarter over quarter, which is exactly why a technical SEO or infrastructure review should treat crawler-access policy as a standing line item, not a settings page you configure once and forget. For a high-traffic ecommerce catalog especially, where search and filter pages are both expensive to serve and heavily hit by exactly this kind of bot traffic, the cost of not auditing this regularly shows up first on the infrastructure bill, and only much later, if ever, in an actual explanation for why it climbed.
The first step, before any policy change at all, is simply reading the logs long enough to know what's actually hitting the site. Most teams reach for a robots.txt edit before they've pulled a single week of raw server logs and sorted by user-agent and request path, which means the fix gets picked based on a guess about what's driving the bill rather than the actual traffic pattern underneath it. An hour spent on that log review, isolating which paths are getting hit hardest and by which named or unnamed crawler, tells you more about where the real cost is concentrated than any general industry statistic in this piece, including the ones cited above. The industry numbers tell you the problem is real and getting worse. Your own logs tell you exactly what to fix first.
It's also worth checking whether your CDN or hosting provider already offers crawler-specific rate limiting or bot-management tooling before building anything custom. A lot of the infrastructure cost described here is solvable at the CDN layer, catching the repeated, cache-bypassing request pattern before it ever reaches origin, rather than requiring a bespoke crawler-classification system built in-house. That's a meaningfully cheaper fix than it sounds, and it's the kind of infrastructure-level lever that pairs naturally with the content-and-citation side of a broader GEO strategy rather than fighting against it.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.