The most common technical mistake we find in AI readiness audits is not a missing file or a broken tag. It is a robots.txt rule that blocks a training crawler, was intended to protect content from being learned, and has quietly been read by everyone involved as protection against being used in AI answers. Those are different crawlers. They are usually operated by the same company, which is exactly why the confusion persists.
AI crawler user agents fall into four categories that behave differently, obey different rules, and produce different consequences when you block them. Sorting them correctly takes about twenty minutes and is the difference between a considered content policy and an accidental one.
The four categories of AI crawler user agents
Here is the taxonomy, with a representative agent from each. The distinction that matters is the middle column: what the visit is for.
| CATEGORY | EXAMPLES | WHAT IT DOES | BLOCKING EFFECT |
|---|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Amazonbot, Meta-ExternalAgent | Collects data for model pretraining | Removes you from future training; no effect on live citations |
| Search and retrieval | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Google-CloudVertexBot | Builds the index engines cite from | Removes you from AI answers; this is the expensive one |
| User-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent | Fetches a page because a person asked right now | Often unblockable; several ignore robots.txt by policy |
| Opt-out tokens | Google-Extended, Applebot-Extended | Not crawlers at all; directives that gate training use | Controls training only; never appears in your logs |
That fourth row causes a specific kind of confusion during audits. Google-Extended and Applebot-Extended make no HTTP requests and generate no log lines. Teams search access logs for them, find nothing, and conclude the directive is not working. It is working; it is simply a policy flag attached to content Googlebot and Applebot already fetched, not an agent that visits.
Training crawlers are not retrieval crawlers
This is the distinction with real money attached, so it is worth being precise. OpenAI runs GPTBot for pretraining and OAI-SearchBot for the index that ChatGPT search cites. Anthropic runs ClaudeBot for training and Claude-SearchBot for retrieval. The two jobs are separable, and the vendors separated them deliberately so publishers could make different decisions about each.
The asymmetry is what makes the mistake expensive. Training exclusion is a long-horizon, low-urgency decision about future models. Retrieval exclusion takes effect on the next answer a buyer reads. One is a policy preference. The other is a revenue channel, and they are one line apart in the same file. We covered the adjacent question of whether blocking is a defensible strategy at all in the case for charging AI crawlers rather than blocking them; this piece is narrower and more mechanical, which is where the errors actually live.
The fetchers that ignore robots.txt on purpose
Category three breaks the mental model most technical SEOs carry, and it is not a bug. When a person asks an assistant to read a specific page, the vendor treats that fetch as the user's action rather than the crawler's, and several vendors document that robots.txt does not apply.
Google's user-triggered fetchers, which include Google-Agent, Gemini Notebook, Google-Read-Aloud, and a handful of others, ignore robots.txt on the stated reasoning that the fetch was requested by a user. OpenAI's documentation indicates ChatGPT-User may do the same. Anthropic's Claude-User honours it. Perplexity's Perplexity-User is documented to honour it, with violations observed in the wild.
“Respecting robots.txt in user-triggered fetchers is a vendor-level policy, not a category-level one. There is no rule you can write that covers them all.”
The practical consequence: robots.txt is a cooperative standard, and cooperation now varies by vendor and by fetch type within the same vendor. If you need enforcement rather than a request, that has to happen at the edge with rate limits, WAF rules, or bot management, not in a text file. And it should be a deliberate decision, because the traffic in this category is the highest-intent traffic in the entire taxonomy: an actual human, right now, asking to read your page.
What AI crawler user agents mean for your robots.txt
There is also a category-five problem: agents that publish no documentation and, in some cases, do not identify themselves at all. ByteDance's Bytespider publishes nothing. xAI's crawler has been observed using spoofed Safari and Chrome strings. Microsoft's Copilot Actions runs with a standard Edge user agent and no bot signal.
You cannot write robots.txt rules for traffic that does not announce itself, which sets a hard ceiling on what the file can do for you. Treat robots.txt as the mechanism for the well-behaved majority and edge controls as the mechanism for everything else. Do not treat either as complete.
This is also why crawl-based estimates of AI interest run low. If a meaningful share of the agents reading your site are masquerading as browsers, your bot analytics undercount the real demand, sometimes badly. Teams then conclude that AI engines are not interested in their content and deprioritize the work, when the more accurate statement is that they cannot see the interest with the instruments they are using. Verified IP checks close part of that gap. Nothing closes all of it.
Two notes on that block. First, it is a template and not a recommendation for every site: publishers with licensing leverage, and anyone whose content is the product, will reasonably choose differently. Second, Google-Extended gates Gemini training without affecting Google Search inclusion or ranking, and Applebot-Extended does the same for Apple Intelligence without touching Siri, Spotlight, or Safari Suggestions. Those separations are the reason the tokens exist, and they are safer to use than most teams assume.
Verifying who is actually visiting
User agent strings are self-reported and trivially forged, so any policy built on them alone is advisory. Three verification methods exist, in ascending order of confidence.
Start with published IP ranges. They are static files, the check is a few lines of code at the edge, and they turn your bot analytics from a report of what visitors claimed into a report of what they were. That single change usually reveals that a meaningful share of what a site recorded as AI crawler traffic was neither AI nor a crawler. It is also the foundation for the crawl-log work that sits underneath any serious technical audit.
The configuration to ship
Four steps, in order, none of which takes longer than an afternoon.
First, read your current robots.txt with the taxonomy in hand and mark every AI rule as training, retrieval, user-triggered, or token. Most audits turn up at least one rule whose category nobody on the team could name. Second, decide training and retrieval separately and write them down as separate decisions, because the moment they are one decision the retrieval side loses by default. Third, verify with IP ranges rather than user agent strings. Fourth, confirm your pages are actually usable once fetched, which is a different problem entirely and the reason AI crawlers that do not render JavaScript remain the most under-diagnosed cause of missing citations.
Then check the file that tells the well-behaved half of this population what you actually want them to read. A current llms.txt does nothing for the crawlers that ignore you and a fair amount for the ones that do not. For teams shipping documentation-heavy products, where the buyer research happens in an assistant rather than a browser, this configuration is closer to a distribution decision than a technical one, which is why we treat it that way for developer tools clients.
Keep the vendor documentation close while you do this, because the strings change more often than the taxonomy does. OpenAI publishes its agent list and IP ranges at openai.com/gptbot, and the equivalent files from Anthropic, Perplexity, Bing, and Common Crawl are all linked from their respective bot pages. Bookmark the JSON files rather than copying the strings into a config you will forget to update. An agent string that drifted six months ago is indistinguishable, from your logs, from a crawler that stopped visiting.
One last check worth running before you consider this closed. Pull thirty days of logs, filter to every agent in the four categories above, and compare crawl volume against the pages you actually care about being cited on. The pattern we see most often is not under-crawling but misdirected crawling: retrieval agents spending their budget on paginated archives, tag pages, and filtered category URLs while the comparison pages and product documentation get visited once a month. Nothing in your robots.txt is wrong in that scenario. The crawl is simply being spent in the wrong place, and the fix is internal linking and sitemap hygiene rather than agent rules.
That is the honest summary of this whole area: the agent taxonomy is the part everyone gets wrong first, and crawl allocation is the part that limits results once they get it right. Fix the categories, verify by IP, then go look at where the crawl budget is actually landing. In that order, the work is a week. In the reverse order, it is a quarter of tuning something that a single misfiled robots.txt line was overriding the whole time.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Josh leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.