We wrote last year about the case for charging AI crawlers instead of blocking them. Cloudflare just proved the first version of that idea wasn't the final one.
From pay-per-crawl to pay-per-citation
Pay Per Crawl, Cloudflare's original metering system, let publishers set a price per fetch and issue an HTTP 402 to bots that wouldn't pay it. It worked as a proof of concept: publishers billed for well over a billion 402 responses a day at points in its first year, and outlets from Condé Nast to the Associated Press adopted it. But billing per fetch has an obvious flaw, one that only becomes visible once you have a year of data: a crawl is not the same thing as value delivered. A bot can fetch your page a thousand times and cite it zero.
Around July 1, 2026, Cloudflare announced the fix: a shift toward compensating publishers when their content is actually cited in a generated answer, not simply when it's fetched. Two launch partners went live with the model immediately. Ceramic.ai runs pay-per-query, with reporting back to publishers on which queries and rankings drove the payment. You.com is running on-demand payments tied to premium content an agent pulls into a session. Both are early, both are opt-in, and both are a direct admission that the crawl-based meter Cloudflare shipped a year earlier was measuring the wrong event.
The distinction is worth spelling out plainly, because it's easy to read "pay-per-crawl" and "pay-per-citation" as two flavors of the same idea. They're not. A per-crawl fee is a toll on access: it charges for the act of showing up at the door, regardless of whether anything useful happens once inside. A per-citation fee is a royalty on outcome: it only pays out when the visit actually produces something a publisher can point to, a specific answer, to a specific buyer, that used their content. One meters traffic. The other meters value. Cloudflare spent a year running the first model at scale before the data made clear it was pricing the wrong side of the transaction.
Why the old model didn't hold
The data explains the pivot better than any press release would. By June 2026, AI training crawls made up 50.6% of all traffic on Cloudflare's network, against just 10.7% for the legitimate search bots that actually send referral traffic back. Separate research from Cloudflare found that more than half of AI bot crawls fetch a page that hasn't changed since the previous visit, meaning a huge share of that 50.6% is redundant by definition, not new discovery.
| TRAFFIC TYPE | SHARE OF CLOUDFLARE NETWORK, JUN 2026 |
|---|---|
| AI training crawls | 50.6% |
| Legitimate search bots | 10.7% |
| Everything else | 38.7% |
Charging a flat per-crawl fee on traffic that's mostly redundant re-fetches was never going to feel fair to publishers footing the infrastructure bill, and it wasn't tightly connected to whether the crawl produced anything a buyer actually saw. Pay-per-citation ties the payment to the outcome that matters, which is the same shift our own reporting work has been pushing clients toward all year: stop counting the crawl or even the mention, and start counting the cited outcome.
Pew Research's finding that users click through only 8% of the time when Google shows an AI summary adds another layer to why the old model felt increasingly disconnected from reality. Even on the rare crawl that does inform a citation, the odds that citation converts into an actual visit to the publisher's site are low. A per-crawl fee was charging for access to a pipeline where the final, human-visible step, the click, was already vanishing. Pay-per-citation at least ties the payment to the step before that one, the step that still reliably happens.
The mixed-use crawler crackdown
The second half of the announcement got less attention but matters just as much. Cloudflare set a deadline to filter out "mixed-use" crawlers, bots that present themselves as search-indexing crawlers, the kind sites have always allowed by default, while quietly feeding the same fetched content into AI training pipelines. Forbes, Engadget, and TechCrunch all covered the crackdown the same week, and the framing across all three was consistent: publishers had been unknowingly consenting to AI training simply by allowing what looked like ordinary search indexing.
The practical effect for most site owners is that a robots.txt allowlist built years ago, back when "search crawler" and "AI training crawler" were the same category of visitor, no longer describes what's actually happening on the other end of those requests. A rule that was written to let Googlebot in may, without anyone updating it, also be letting through a crawler whose only stated purpose used to be search indexing and has since expanded, quietly, into feeding a model training pipeline. Cloudflare's filter is effectively forcing publishers to re-declare consent for a use case that changed underneath them.
What this means for machine access
None of this changes the fundamentals we've been arguing since the anatomy of an AI citation: being reachable, structured, and credible is still the price of entry to any AI answer. What it does change is the economics sitting underneath that access. A publisher on the pay-per-citation model has a direct financial reason to make sure their most citable pages are actually reachable and structured well, because now a clean citation has a dollar value attached to it, not just an abstract visibility benefit.
It also raises the stakes on the crawler-blocking decisions we've argued against making by country or by default. A publisher who blindly blocks all AI bots to avoid the mixed-use problem risks losing the pay-per-citation upside along with the training-data exposure. The more precise move, letting declared crawlers in, filtering the ones misrepresenting themselves, and getting paid when cited, is exactly the kind of nuance Cloudflare's new stack is built to support, and it changes the crawl-to-referral math we walked through in our economics piece on AI crawl traffic: a crawl that used to be pure cost now carries a real path to revenue attached.
What to do about it
There's a broader lesson here beyond Cloudflare specifically. Every metering model built for AI traffic so far has been a first draft, corrected once real usage data revealed what it actually priced. Pay Per Crawl priced access; the correction showed access wasn't the scarce resource, citation was. The next correction, whenever it comes, will likely price something even further downstream, actual referred revenue, or a completed agent transaction, once enough data exists to show that citation itself is still one step removed from the value publishers are chasing.
It's worth being clear-eyed about how early this still is. Two launch partners, opt-in enrollment, and a pricing model that's still being worked out in public isn't a mature market yet, and enterprises shouldn't restructure a content strategy around a system this new. What it does deserve is attention and a pilot, not a wait-and-see posture, because the publishers who join early tend to shape how models like this get priced and reported once they scale, the same first-mover dynamic that decided who won the early Pay Per Crawl adoption wave.
If you're on Cloudflare, check whether pay-per-citation is available for your plan and whether your highest-value pages qualify. If you're not, treat this as a preview of where every major CDN and hosting platform is heading: crawl-based metering was a first draft, and citation-based compensation is what replaces it. Either way, the underlying work doesn't change. Confirm your machine access is clean through a real technical audit, keep your best pages structured for extraction, and build citation rate, not crawl volume, into how you evaluate your GEO program. We're already wiring citation-based economics into the access-layer work we do for clients like the cybersecurity platform we took from blocked to consistently cited, and it's the same access-economics dimension we score in our enterprise AI agent readiness framework: what it costs you, in dollars and crawl budget, to stay reachable as this shifts from metered access to paid outcomes.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.