Adweek reported on August 5 that publishers are pulling back from Google's AI search while creators push further in. Buried in that story is the detail that actually matters for every enterprise site, not just media companies: Cloudflare's crawler-blocking software, already running at the Financial Times, Condé Nast, and The Atlantic, starts blocking crawlers used for both search indexing and AI training on September 15. Not crawlers used for one or the other. Crawlers used for both, at the same time, with no clean way to keep one function and drop the other. If you want to block AI crawlers without also blocking the search bots your rankings depend on, the work starts now, not on September 15.
Why the deadline to block AI crawlers just got real
We've covered pieces of this story before it had a face. Our July 24 piece on Cloudflare's pay-per-citation shift tracked the economics: AI training crawls were already 50.6% of Cloudflare's network traffic against 10.7% for legitimate search bots. Our August 4 piece on the default-block mechanism walked through exactly how the September 15 flip works: sites that block Training-category crawlers get Googlebot blocked as a side effect too, because Google runs one crawler for both search indexing and Gemini training, and Cloudflare applies the most restrictive rule to any bot that serves more than one purpose.
Both of those pieces described a policy. This is the week it stopped being one. Adweek's report names three publishers who have moved from "eligible for this default" to "actually running it": the Financial Times, Condé Nast, and The Atlantic. That's not a hypothetical CDN setting anymore. It's a live production decision at three organizations with enough legal and technical resources to weigh the tradeoff carefully, and they weighed it in favor of blocking. Adweek frames the story as a crack forming between Google's search crawler and major publishers, because Cloudflare's policy won't let a publisher say "index me, don't train on me" for any crawler that does both. You either accept both or block both.
Here's why that matters if you're not a publisher. The FT, Condé Nast, and The Atlantic have technical teams whose entire job is protecting content value and search visibility at once. If they're opting in anyway, it tells you the tradeoff has already cleared a real risk bar for organizations that can't afford to get it wrong. Most enterprise marketing and IT teams do not have a standing crawler review process. September 15 is five weeks out from today. That's enough time to run a proper machine access audit if you start now, and not enough time if you wait for a traffic drop to tell you something changed.
“The deadline didn't change. What changed is who's actually pulling the trigger on it.”
Search-indexing, AI-training, and hybrid crawlers, explained
The reason this deadline is confusing is that most robots.txt files and CDN rule sets were written for a world with two categories of bot: the ones you want (search) and the ones you don't (spam, scrapers). AI crawlers broke that binary into three categories, and the third one is where nearly every accidental block or accidental miss happens. Before you touch a single rule, get clear on which bucket each crawler hitting your site actually belongs in.
| CRAWLER TYPE | EXAMPLE USER AGENTS | WHAT IT DOES | DEFAULT RULE MOST TEAMS SHOULD SET |
|---|---|---|---|
| Search-indexing | Googlebot (search function), Bingbot | Crawls to build a search index that can send you ranked, referral-generating traffic | Allow |
| AI-training | GPTBot, CCBot, Google-Extended, Bytespider | Crawls to feed a model's training set, with near-zero referral traffic back | Block or meter, per your policy |
| Hybrid / multi-purpose | Googlebot (AI grounding), ClaudeBot, PerplexityBot | Same crawler serves both search-adjacent citation and model access, no separate user agent to isolate the two | Decide deliberately, don't default |
That table is a practical reference for how to think about the split, not a data claim, but the category boundaries are the real ones Cloudflare's policy is built around. The first row is easy: nobody serious argues for blocking the crawler that puts you in front of buyers. The second row is where most of the actual dollar savings sit, since training crawlers historically return next to nothing in referral traffic for the crawl budget they consume. It's the third row that decides whether September 15 costs you anything. ClaudeBot and PerplexityBot increasingly function as retrieval crawlers for live citations, not just training scrapers, which means blocking them the way you'd block a pure-training bot can cut you out of an AI Overview or a Perplexity answer, not just a training set. Our piece on the difference between blocking AI crawlers by accident found 30% of sites had already locked out GPTBot through stale rules nobody had reviewed in years. Hybrid crawlers turn that same carelessness into a search-visibility problem, not just a training-exposure one.
Five plays to block AI crawlers before September 15
This is the same sequence we run for clients doing a technical SEO audit under a hard deadline. Five plays, run in order, each with a clear stopping point so you know when you're actually done instead of just tired of looking at logs.
What happens if you do nothing
Nothing happens to sites that were never going to touch their crawler defaults anyway. Everything happens to sites that get swept into a policy flip they never reviewed. That's the pattern in every version of this story we've covered this year: not a deliberate decision going wrong, but a default nobody looked at doing the deciding instead. We found the same failure mode in JavaScript-rendering blind spots, where sites looked fine to a human and returned an empty shell to a non-hydrating crawler, and nobody noticed until a citation number dropped and someone finally went looking for why.
The specific risk here is narrower but sharper. If your site sits behind Cloudflare or a comparable CDN and someone, at any point, enabled a blanket "block AI crawlers" toggle to stop training scrapers, that toggle may already be catching hybrid crawlers you'd actually want reaching you, the ones increasingly responsible for live citations in ChatGPT, Perplexity, and AI Overviews, not just training sets. September 15 raises the stakes on that existing exposure because it's the date the industry's most cautious, best-resourced publishers confirmed they're comfortable accepting the tradeoff. Their calculus, protect content value even at some search-visibility cost, is not automatically your calculus. Most enterprise sites make more money from being findable than they lose from being trained on. That argues for the opposite default: allow search and cited-retrieval traffic, block pure training, and stop treating block AI crawlers as one undifferentiated switch.
Adweek's report on the September 15 deadline is worth reading in full if you want the publisher-side context on why FT, Condé Nast, and The Atlantic made the call they did. But their decision doesn't answer the question for you. Yours is a different site, a different traffic mix, and very possibly a different AI-crawler bot list than a media publisher's. The only way to know your own answer is to run the audit, not borrow theirs.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.