Somebody on your team has already run this audit once, probably more than once. The site looks sharp. Google indexes it without a complaint. Core Web Vitals are green. And yet when a prospect asks ChatGPT or Perplexity a question your product answers well, you're nowhere in the response — not buried at the bottom, just absent, while a competitor with a worse Lighthouse score gets quoted by name. Someone checks robots.txt to make sure nobody's blocking the crawler. Nothing's blocked. The mystery gets filed away as unsolvable and everyone moves on.
I'd normally drop in the exact Reddit thread where a technical lead is losing it over this in real time, because that thread absolutely exists somewhere in r/TechSEO. Reddit's been unreachable for this research for weeks running now — still down again as of today, a persistent outage at this point, not a one-off — so this one comes from a site audit instead of a comment section. Same complaint, minus the username. It's the exact frustration that would be sitting at the top of that subreddit if we could still read it.
Here's the part that isn't actually a mystery once you say it plainly: a lot of AI crawlers fetch your JavaScript file and never execute it. Not "execute it slowly." Not "execute it with limitations." Never. If your pricing table, your product specs, or your comparison content only render after the browser runs client-side JavaScript, an AI crawler that skips execution is reading an empty shell. Not your page. A placeholder for your page.
Why Your JS-Heavy Site Isn't Getting Cited Anywhere
The instinct is understandable. Something's wrong, so you check permissions first. Are we blocking anyone? Did a security plugin write a rule nobody remembers? That's a real failure mode, and we've covered what accidental crawler blocking actually costs a site in detail elsewhere — but it's the wrong first question for a huge chunk of teams, because their robots.txt is clean. Nobody's blocking anything. The crawler shows up, gets waved through, and still leaves with nothing usable.
That's a different failure than a permissions problem, and it's a quieter one. A React or Vue app built with client-side rendering typically ships a near-empty HTML skeleton first, then fills it in once the browser downloads and runs the JavaScript bundle. A human's browser does that in under a second and nobody notices. A crawler that fetches the same HTML and skips the execution step gets the skeleton. Forever. It never sees the fill-in, because the fill-in was never its job to trigger — that's the browser's job, and the browser never showed up.
This is a distinctly different problem from the one everyone's been arguing about lately, which is whether an llms.txt file actually does anything. That debate assumes the crawler can already read your content and is just deciding whether to prioritize it. Rendering is the layer underneath that argument. It's not about instructions or prioritization. It's about whether there's anything on the page at all by the time the crawler stops looking.
AI Crawlers Download Your JavaScript. They Don't Run It.
The most direct evidence on this comes from Lantern's crawler analysis, a piece by Collins titled "AI Crawlers Do Not Render JavaScript," first published March 23, 2026 and updated June 11, 2026. Working from a large-scale Vercel crawl-behavior analysis covering more than 500 million GPTBot fetches, Lantern found zero evidence of JavaScript execution across that entire sample. Not rare execution. None detected.
What makes that finding land harder than a simple "AI bots ignore JS" headline is the detail sitting right next to it: GPTBot downloads JavaScript files roughly 11.5% of the time. It isn't skipping the files. It's requesting them, receiving them, and then doing nothing with what's inside — like a mail carrier who delivers a sealed envelope and files a delivery receipt without anyone ever opening it. The file exists on the crawler's side. The content inside it functionally doesn't.
That 44.2% figure compounds the rendering problem instead of sitting next to it. If nearly half of ChatGPT's citations are pulled from the top third of a page, and your top third is a loading skeleton because the real content only paints in after hydration, you're stacking two disadvantages on top of each other — content that arrives late, on a system that already prefers content that arrives early. This hits developer-tools and SaaS sites especially hard, since that's exactly where the heaviest client-side frameworks tend to live; we go deeper on the mechanics for JS-heavy developer tools sites specifically and walk through the full fix in our guide to technical SEO for headless and JavaScript sites, which is worth the longer read if this is the first time you're hearing any of this.
Blocking AI Crawlers Is Rare. Invisibility Is the Norm.
Website Auditor's AI Visibility Index, published by Armstrong HoldCo LLC, backs up the same pattern from a different angle. The second edition, out August 2, 2026 (the first ran July 31, 2026), covers 531 audits across 458 domains in 38 sectors. Only 8.9% of audited sites block at least one AI crawler via robots.txt. Meanwhile, 94.8% of audited sites never appear at all in AI assistant answers — a figure built from checking 193 sites against 5,978 answers across ChatGPT, Claude, Gemini, and Perplexity between July 23 and August 2, 2026, with Gemini posting the highest citation rate of the four at just 2.9%.
Sit with the gap between those two numbers for a second. Blocking, at 8.9%, is the exception. Invisibility, at 94.8%, is close to universal. Whatever's driving that gap, it is not primarily a permissions problem — there aren't enough blocked sites to explain a number that high. Something else is eating the difference, and unrendered JavaScript is one of the biggest, least-discussed candidates.
| WHAT TEAMS CHECK | WHAT THE DATA SHOWS | WHAT IT ACTUALLY MEANS |
|---|---|---|
| Robots.txt / crawler blocking | 8.9% of audited sites block at least one AI crawler | Blocking is the exception, not the explanation |
| Appearance in AI assistant answers | 94.8% of audited sites never appear at all | Invisibility is the default outcome, not a failure state |
| Best-performing engine in the sample | Gemini cites at 2.9% — highest of the four tested | Even the most generous engine is still citing a small minority |
One honest caveat worth flagging plainly: the study's own authors describe the sample as self-selected, not random, and skewed toward small and mid-sized businesses that are already doing some AI visibility work. That means the real invisibility rate across the broader web, including sites doing nothing at all on this front, is plausibly higher than 94.8%, not lower. Take the direction of the finding seriously and hold the exact number loosely.
None of this is the same story as the llms.txt debate, and I'd rather say that outright than let the two blur together. A permissions file can't fix a rendering problem. If the crawler never executes the JavaScript that produces your content, telling it politely where to look changes nothing about what's waiting there when it arrives.
The Conversion Gap Hiding Behind a Blank Page
There's a real business number sitting inside Lantern's dataset that deserves more attention than it's getting: visitors referred by AI assistants convert at 14.2%, against 2.8% for visitors arriving from a standard Google click. That's not a rounding difference. Whoever an AI engine sends you, they've already been pre-sold by an answer that quoted your content directly, which is exactly why it stings when the crawler that would have delivered that traffic never got past your loading skeleton in the first place.
The content-type breakdown in the same dataset sharpens the stakes further. Product pages earn 20.1% of all AI citations in Lantern's dataset, well ahead of comparison content's 9.3% — and product pages are frequently the exact pages built as client-side-rendered components, because that's where teams want dynamic pricing, live inventory, or interactive configurators. The pages worth the most in citation share are disproportionately the pages most likely to be invisible to a crawler that skips execution.
That last data point is the one I'd sit with longest. Prerendering didn't require a content strategy overhaul or a rewrite of the product pages themselves. It required making the existing content visible to something that was already trying, and mostly failing, to read it. The demand from AI crawlers was already there. The page just wasn't answering the door.
The Test You Can Run This Week
You don't need a vendor contract to find out whether this applies to you. You need twenty minutes and a page you actually care about — pricing, a flagship product page, a comparison page you've invested real content into.
It's a classic test, and it's still shockingly underused — most marketing teams have genuinely never tried it, because nothing about their day-to-day workflow ever prompts them to. Robots.txt gets checked because someone remembers to check it. Rendering gets skipped because nobody's ever been burned by it in a way they could trace back to a cause.
If that test turns up a real gap, the fix usually isn't a rewrite. It's server-side rendering, static generation, or prerendering for the specific pages carrying your highest-value content — which is exactly the kind of diagnostic work our SEO and GEO audits are built to catch before it costs you another quarter of citations you were never in the running for.
Your site can look perfect to every human who visits it and still be functionally blank to the systems increasingly deciding who gets recommended. That's not a permissions story, and it's not an llms.txt story. It's a rendering story, and it's fixable the moment you stop assuming that because Google can see it, everyone else can too.
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.
Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.