Ask most marketing or dev teams whether they're blocking AI crawlers and you'll get the same answer, fast: no, why would we be? Nobody wrote a rule to keep GPTBot out. Nobody flipped a switch on purpose. That confidence is exactly the problem. An independent crawl of 1,500 live websites found that roughly 30% of sites are blocking AI crawlers right now, and most of the people running those sites have no idea it's happening. Not a strategic call. Not a security team weighing tradeoffs. Just an old rule nobody remembers writing, still doing its job.
Worth being specific about where this comes from, since we like to name our sources. Reddit's usual homes for this exact conversation, r/TechSEO and r/SEO in particular, were unreachable during this run's research pass. So the discovery thread we're pointing to instead is Hacker News: a submission from user aggeeinn titled "I crawled 1,500 sites: 30% block AI bots, 0.2% use llms.txt," linking to WebsiteAIScore's 1,500-site audit. Same shape of discourse, different platform. The numbers below come straight from the linked report, not the comment thread.
The audit crawled all 1,500 sites with a custom crawler built to behave the way GPTBot and its peers actually behave, not the way a polite, JS-rendering search bot behaves. That distinction is the whole point of the exercise. These are AI crawlers being tested as AI crawlers, and a site that passes every check in Google Search Console can still be blocking AI crawlers entirely, because nothing in that dashboard was ever built to tell you.
Nobody sets out to block AI crawlers
The 30% number is the one that should stop you mid-scroll, because it isn't describing sites run by careless teams. It's describing sites run by teams exactly like yours: ones with a robots.txt file that's been sitting untouched for years, holding a rule that made sense at the time and never got revisited. That's how you end up blocking AI crawlers by accident. Not through a decision. Through neglect of a file most people only touch once and then forget exists.
The pattern the audit turned up is boring in the specific way that makes it believable. A security plugin installed years before GPTBot existed, set to block any user agent it doesn't recognize by default. A robots.txt rule copied from an old template or a competitor's site, disallow-all except a couple of paths, never audited since. A CDN-level bot-management rule tuned to stop scrapers, applied broadly enough to catch OpenAI's, Perplexity's, and Anthropic's crawlers in the same net. None of it was aimed at AI crawlers specifically. All of it hits them anyway.
What 1,500 sites turned up
Robots.txt isn't the only way to disappear from an AI crawler's view, and it might not even be the bigger problem. 40% of the marketing sites in the audit leaned so heavily on client-side rendering that they showed up as close to blank pages to any crawler that doesn't execute JavaScript. That's not a robots.txt problem. That's an architecture problem: the page is technically reachable, technically allowed, and still invisible, because the content only exists after a script runs, and most AI crawlers don't run it.
Most AI crawlers, GPTBot included, fetch raw HTML and move on. They don't sit around waiting for your JavaScript framework to hydrate a page, build the DOM, and paint the content a browser would eventually show a person. If your homepage or your pricing page ships as an empty div that fills in after load, a non-hydrating crawler sees the empty div and stops there. It's the same failure mode we've mapped out in the technical differences between AI crawlers and AI agents: the crawler and the eventual human visitor are looking at two completely different pages.
| CRAWLER | BELONGS TO | WHAT BLOCKING IT COSTS YOU |
|---|---|---|
| GPTBot | OpenAI | No inclusion in ChatGPT's training or retrieval-backed answers |
| PerplexityBot | Perplexity | No citation eligibility in Perplexity's answer engine |
| ClaudeBot | Anthropic | No inclusion in Claude's web-grounded responses |
| Google-Extended | Opts you out of Gemini and AI Overviews training and grounding, separate from classic Googlebot |
That's the practical cost behind the 30% number. It isn't abstract lost "authority." It's these specific bots, with these specific names, quietly turned away at the door by a rule that predates most of them.
Three checks to see if you're blocking AI crawlers
None of this requires a developer sprint. The audit above is a five-minute check most teams have simply never run, because until recently there was little reason to think about non-human, non-JS-executing visitors at all. It's also usually not the marketing team's fault when the block turns up. The rule was more likely written by whoever set up the security plugin, or whoever configured the CDN two jobs ago, and neither of them was thinking about AI crawlers because AI crawlers didn't exist yet. Here's the version you can do today, no tools required beyond a browser and your own robots.txt file.
The first two checks matter more than the third, and it's worth being honest about why. Robots.txt and rendering problems are access problems, they determine whether a crawler can reach your content at all. llms.txt is closer to a courtesy note left for a visitor who already got in the door. Fix the door first.
Where llms.txt actually fits
We've been careful about this one, on purpose. Whether llms.txt actually moves AI citations is a question we already ran the data on, two independent studies covering a combined 337,894 domains, and neither found a measurable citation lift from having the file. So we're not going to tell you it's the fix here. It isn't.
What it is: a file that costs nothing to add, doesn't conflict with anything else on your site, and puts you in a group of roughly 0.2% of sites that have bothered. That's not a growth lever. That's cheap insurance against a future where the format matters more than it does today, bought for the price of a text file and a few minutes. Ship it after you've fixed robots.txt and rendering, not instead of fixing them.
If your site is JavaScript-heavy by design, a full rebuild usually isn't the answer, server-side rendering or a rendering fallback for known crawler user agents usually is, and it's a fix worth building in from the start rather than retrofitting two years later. It's the same lesson we walk through in technical SEO for headless and JavaScript-heavy sites, and the same principle behind our work with MarketCheck, where getting the rendering layer right mattered more than any single meta tag. If you're building something new, it's worth baking crawlability in from day one through our site design and development builds rather than auditing your way out of it once the traffic gap shows up in the data.
The three-check audit: do this today
You don't need a quarter-long AI-readiness initiative to fix this. You need the same five minutes the 1,500-site audit spent on each of the sites it found broken, pointed at your own domain instead.
Run all three today, not because any one of them guarantees a citation, but because 30% of a random sample is currently failing the first one without knowing it. If your audit turns up something bigger than a stale rule, an entire site built on client-side rendering, or a bot-management layer nobody on your team fully understands anymore, that's a longer conversation, and it's the kind of conversation our SEO services exist to have. But you don't need us for the five-minute version. You just need to actually run it.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.