You Unblocked GPTBot. That Was Never the Hard Part.

Every technical SEO audit I've read this year has the same line item, usually filed under "AI readiness": unblock GPTBot, PerplexityBot, and ClaudeBot in robots.txt, ship an llms.txt, mark the section done. It reads like diligence. It reads like the AI crawler question got handled. What it actually handled was a much smaller problem than the checklist implies, and the evidence has been sitting in plain server logs the whole time.

I'll say upfront where this one came from, because it matters. The sharpest practitioner discussion on this exact question wasn't on Reddit this week — the usual subreddits weren't reachable when I went looking, so this runs on the closest real discourse I could verify instead: a widely-cited Hacker News thread, still being referenced and argued over months after it was first posted. It's arguably a better source than most vendor blogs on this exact topic, because it's server logs and working engineers arguing from evidence, not marketing copy. And it goes straight at a gap most technical SEO audits still treat as one bucket: the difference between a bot that trains on your site and a system that's live, in front of a user, trying to answer a question with it.

The AI crawlers vs. AI agents mix-up everyone's making

Here's the mental model baked into almost every AI-readiness checklist I've seen this year. Somewhere there's a bot. It has a name — GPTBot, ClaudeBot, PerplexityBot — and a line in robots.txt either lets it through or doesn't. Unblock the names on the list, ship an llms.txt file pointing at your best pages, and you've handled "GPTBot access" as a category. One box, one owner, one status: done.

The problem is there's no single "it" here. There are at least two very different systems reading your pages, built by different teams, for different jobs, on completely different schedules. One is a bulk scraper hoovering up the open web to build a training corpus. The other is a live retrieval agent, acting on behalf of one specific person, mid-conversation, right now. Confusing them is like designing one onboarding flow for a delivery truck and a customer walking in the front door, because both of them technically "came through the entrance." We've written before about what llms.txt actually is and isn't, and the short version is that it was built with a specific reader in mind — one that, per the server-log evidence below, mostly isn't the reader people assume.

What the server logs actually show

The Hacker News thread that's been circulating started with someone doing the boring, useful thing nobody does often enough: pulling actual server logs across several sites and checking who really requested llms.txt and AGENTS.md files. Not who was supposed to. Who did. The answer, across the sites checked, was nobody — at least not the crawlers you'd expect. No GPTBot hits on those paths. No ClaudeBot. No PerplexityBot. The only requests that showed up came from generic scrapers and cloud-platform bots with no relationship to any AI lab at all.

The top comments on that thread converged on the same explanation, and it isn't flattering to how the industry talks about this. The crawlers that build LLM training data are, for the most part, basic regex-based scrapers. They're not reasoning about your site. They're not deciding your llms.txt file looks trustworthy and worth following. They pull HTML, follow links, and move on, the same unglamorous way search crawlers have worked for two decades. A directive file sitting quietly at the root of your domain, asking politely to be noticed, assumes a kind of judgment these systems were never built to exercise.

That doesn't mean llms.txt does nothing for anyone, and several commenters were specific about the case where it does work: an individual LLM client agent — your own Claude or ChatGPT session, fetching a docs page on demand mid-conversation because you asked it to look something up. That's a narrow, user-driven, real-time use case, and it's a genuinely different animal from a bulk crawler indexing the web for training. What actually happens once a model starts processing a page looks nothing like a training crawl, and the confusion between the two is most of what's driving bad advice on this topic.

One more detail from that thread is worth sitting with. A few commenters noted that some LLMs reportedly fail to parse certain legacy markup patterns, like old-style HTML frameset content, cleanly at all. That's a small, specific data point, but it points at something bigger: a lot of these systems are working from a rendered version of your page in many contexts, not a careful read of your raw source or your directive files. If your technical SEO for AI search plan assumes a careful reader parsing your intentions, you're planning for a reader that mostly isn't there.

That log-level finding lines up with the adoption data, too. SE Ranking's crawl of 300,000 domains found llms.txt sitting on just over one in ten sites, with no measurable link between having the file and getting cited more often. We went deep on that specific dataset in a separate piece on whether llms.txt actually works, so I won't re-run the argument here. The short version for this piece: the adoption number and the server-log finding are two different pieces of evidence pointing at the same conclusion from two different angles.

AI agents are a different audience, with a different job

Now hold that finding next to what Mike King, founder of iPullRank, laid out in Search Engine Land on May 29, 2026. His argument, in a piece called "Beyond RAG: Why every AI search platform is now agentic," is that ChatGPT Search, Google AI Mode, Perplexity, Claude, and Gemini have all moved past a single retrieve-and-summarize step. They now run what he calls agentic RAG: plan, retrieve, read, and re-retrieve, in a loop, inside a single user query.

Sit with what that means for one page on your site. It's not evaluated once, at crawl time, by a bot deciding whether to include you in an index. It's potentially evaluated several times, in one conversation, by a system deciding whether to trust this specific paragraph enough to quote it, then circling back to check something it wasn't sure about, then deciding again. That's a fundamentally different bar than "was the page reachable." A page can be perfectly crawlable and still fail every one of those re-retrieval passes, because the answer it needs isn't sitting in one clean, self-contained chunk the agent can lift without stitching together five different sections. That's the actual content of agentic RAG SEO, and it has almost nothing to do with robots.txt.

The audit split: AI crawlers vs. AI agents, checked separately

Put the two findings together and the fix is less about learning a new tactic and more about admitting the old checklist was measuring one thing and reporting it as two. Bulk crawl access and agentic retrieval quality are separate technical requirements, tested with separate methods, and a page can pass one while failing the other completely. A real technical audit has to check both, on purpose, instead of letting "robots.txt looks fine" stand in for the whole story.

None of this is an argument against unblocking GPTBot, PerplexityBot, or ClaudeBot in robots.txt — leave that step in, it's still the floor. It's an argument against believing that step is the ceiling. We've made a version of this point before about teams that block AI crawlers reflexively instead of thinking through what they actually want those crawlers to do, and it rhymes with the mistake here: one bot-facing decision standing in for a much bigger strategic question. The same gap shows up for sites that render their content client-side — reachable in theory, unreadable in the way that actually matters, for a completely different reason than the one most audits check.

So here's the actual next step, and it's a short one. Take your last AI-readiness audit and find the line that says robots.txt or llms.txt. That line answered a reachability question for a scraper that mostly can't act on directive files anyway. Now go run the second test nobody ran: ask Claude, ChatGPT, or Perplexity three real buyer questions, watch what it retrieves and re-retrieves from your site mid-answer, and see whether any single page holds a complete answer on its own. One checkbox was never going to cover both audiences. Split the audit, and you'll finally know which one you actually passed.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.