The llms.txt file most AI crawlers still ignore

Ever since llms.txt showed up as a proposed standard, the debate has run in circles: does anyone actually read it, or is it a placebo for anxious marketing teams? A 12-week crawl study finally answers the question with server logs instead of opinions, and the answer depends entirely on which bot you're asking about.

What the test actually measured

EZY Research placed a valid llms.txt file on 83 real sites between April 27 and July 19, 2026, then logged every server hit from named AI crawler user agents against both llms.txt and robots.txt on the same domains, over the same twelve-week window. Comparing the two files on the same sites at the same time isolates the question cleanly. Crawlers already fetch robots.txt as a matter of course, before they touch anything else on a domain, so the robots.txt count is the baseline for how often a given bot actually visits a site at all. The llms.txt count then shows what share of those visits also touch the new file.

That framing matters more than a raw llms.txt count on its own would. A crawler that reads llms.txt twice out of two total visits to a site is behaving in a completely different way than one that reads it seven times out of four thousand visits. Raw counts without that context are exactly how the llms.txt debate has stayed unresolved for so long: a handful of anecdotal fetches get read as proof the file works, without anyone checking what share of total crawler activity those fetches represent.

The 83 sites spanned a mix of content types and sizes, deliberately not concentrated in one vertical, so the counts below are a reasonable read on typical crawler behavior rather than a result specific to one kind of site.

This is also a more honest test than most of the anecdotal evidence that's circulated on both sides of the llms.txt argument. A single site owner checking their own server logs and finding one or two llms.txt hits over a few weeks can't tell whether that's typical or a fluke, because they have no baseline to compare it against and no visibility into what happened on anyone else's domain. Running the same file on 83 sites at once, logging every crawler by name, and comparing against each site's own robots.txt traffic turns a pile of anecdotes into something closer to a controlled measurement.

It's worth being specific about what counts as a crawler hit here, too. The study logged requests to the literal /llms.txt path from a named, verified AI crawler user agent, not from generic bots, scrapers, or monitoring tools that happen to check every file on a domain out of habit. That distinction matters because a looser methodology, one that counted any hit to the file regardless of source, would inflate every number in this table and make the gap between engines look smaller than it actually is.

The bot-by-bot gap

Three of the four major crawlers treat llms.txt as effectively invisible relative to how often they're already on the site. OpenAI's and Anthropic's bots read robots.txt on nearly every pass, exactly the way a compliant crawler is supposed to, and then largely skip the file sitting right next to it. PerplexityBot didn't touch it once, across all 83 domains, over three full months.

That's a meaningfully different finding than the file being read by nobody at all. It means it's read by almost nobody, on three of the four crawlers that matter most to a GEO program, and the fourth is the exception rather than a sign the other three are catching up.

It also breaks the debate out of a single number. "Does llms.txt work" treats the file as one bet with one outcome, when this data shows four bets running at once with four different outcomes, and only one of them paying off. A team that reads the aggregate 97%-never-read figure and concludes the file is worthless is making the same mistake as a team that reads one anecdotal server log showing a fetch and concludes the file is essential. Neither conclusion survives contact with the bot-by-bot breakdown.

The bars above are scaled for visibility, not literal percentages; ClaudeBot's real figure is 0.3% and GPTBot's is 0.2%. The point the chart makes is the shape of the gap, not the decimal. Three bots round down to nothing. One bot doesn't.

Why some crawlers ignore the file entirely

This lines up with a separate, larger-scale finding. Ahrefs analyzed 137,210 domains and found that among the roughly 38,000 with a valid llms.txt file, 97% received zero requests for it, bot or human, across a full month. The mechanism is straightforward. Training crawlers and bulk indexing systems are built to walk a site's normal structure at scale using well-established signals: sitemaps, robots.txt, internal links. A new, non-standard text file sitting outside that structure only gets picked up if a crawler was specifically engineered to look for it, and most of the largest crawlers, per this data, weren't, at least not as of this test window.

Live-retrieval systems are a different case in principle. When an engine fetches a specific page in response to a specific prompt, rather than crawling in bulk ahead of time, it has more reason to also check a file that claims to summarize the whole site in one place. But the OpenAI and Anthropic figures in this study cover exactly that kind of on-demand bot traffic, not just bulk training crawls, and the read rate is still under half a percent for both. Whatever internal logic decides when ChatGPT or Claude fetch a page during a live answer, checking llms.txt first is apparently not part of it yet.

The one outlier, and what it tells you

It's worth taking seriously rather than dismissing. Meta's crawler reading llms.txt 193 times against 172 robots.txt fetches means it isn't just occasionally checking the file, it's treating it as a routine part of how it evaluates a site, more consistently than it checks the file every crawler is supposed to read first. That's a genuinely different crawling strategy from the other three, and it's the strongest evidence in this dataset that at least one major lab has built llms.txt into its standard retrieval logic rather than leaving it as an afterthought.

It's also a reminder that "AI crawlers" is not one behavior. Four named bots in the same study produced four different patterns, ranging from zero engagement to something close to full adoption. Any blanket claim about whether llms.txt "works" is averaging across engines that are not remotely aligned on how they use it, which is exactly the trap the raw 97%-never-read headline number falls into if you stop there.

It's not clear yet whether the other labs will converge toward Meta's behavior over time, or whether Meta's crawler is simply built differently for reasons specific to its own retrieval pipeline. Either is plausible. What isn't plausible, given this data, is planning a GEO program today around the assumption that OpenAI or Anthropic's crawlers are quietly reading llms.txt in the background the way Meta's is. The logs say otherwise, and logs are the only evidence in this argument that isn't a guess.

Should you still publish it

Yes, with the expectations reset. The cost of shipping an llms.txt file is close to zero: an afternoon to write, no ongoing maintenance beyond keeping it current as pages change. The mistake is treating it as machine access infrastructure on the same level as a clean robots.txt, or as a substitute for not blocking AI crawlers outright, which this same data shows the major bots are actually reading on nearly every visit, llms.txt or not. It's a low-cost bet with a real but narrow payoff, concentrated in whichever crawler happens to be built to check for it this quarter, not a channel you can plan a content strategy around.

There's a version of this that's worse than skipping the file entirely: publishing it once at launch and then letting it slowly drift out of date as the site changes underneath it. A stale llms.txt pointing at pages that moved, or a summary that no longer matches the current product, is worse than no file at all for the one crawler in this dataset that's actually reading it consistently, because it's actively feeding that crawler wrong information instead of simply being ignored. If you ship it, put it on the same content-review cadence as your sitemap, not a one-time setup task you check off once and forget about for a year.

None of this changes the more consequential finding underneath it: every crawler in this study, llms.txt-reader or not, showed up and fetched robots.txt reliably, on nearly every single visit. That's the file actually gating whether any of these bots can reach your content in the first place, and it's worth a full audit before llms.txt gets anywhere near the top of a GEO team's priority list this quarter.

# Your Company
> One-sentence description of what you do.
 
## Core pages
- Product: /product
- Docs: /docs
- Pricing: /pricing
 
## Key facts
- Founded, category, notable customers

Ship it for the B2B SaaS and developer-tool sites most likely to run into the one crawler in this study that actually reads it, keep the file current, and put the bulk of your real effort into the things every crawler in this dataset touches reliably: a clean, unblocked robots.txt, and content structured well enough to be worth citing once a bot actually finds it. The full dataset behind these numbers, including the per-site breakdown and the methodology notes, is published directly by EZY Research for anyone who wants to check the raw logs rather than take a summary's word for it.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.