Technical SEO for headless and JavaScript sites, from scratch

For twenty years, technical SEO rested on a quiet assumption: the HTML your server sent was the HTML the crawler read. Headless architectures and single-page apps quietly deleted that assumption. Now the page a user sees is assembled by JavaScript in the browser, after the initial response, and whether a crawler ever sees the finished page depends on choices your engineering team made for reasons that had nothing to do with search. This guide rebuilds technical SEO for that reality, and for the newer reality sitting on top of it: AI engines that have to render, read, and quote you before you can appear in an answer.

Why headless breaks the old assumptions

In a traditional monolith like WordPress or a Rails app, a request for a page returns a complete document. Every heading, paragraph, link, and canonical tag is present in that first response. A crawler can read it in one pass with no JavaScript engine at all. Headless inverts this. The first response is often a near-empty HTML shell, a bundle of script tags, and a div with an id of root. The words, links, and metadata only materialize after the browser downloads several hundred kilobytes of JavaScript, executes it, calls one or more APIs, waits for the responses, and paints the result. Nothing a crawler cares about exists in the document it first receives.

This is not a hypothetical failure mode. In our audits of headless migrations, the single most common finding is a site that renders perfectly for humans and returns a body with almost no readable content to a crawler that does not execute scripts. The team ships it, traffic slides over the following weeks, and nobody connects the two events because the site looks fine in every browser they test. The gap between what a browser shows and what a crawler ingests is invisible unless you go looking for it, and most teams never do until rankings have already moved.

The stakes rose again with generative engines. Googlebot has run a modern rendering engine for years and can, under the right conditions, execute your JavaScript. Most AI crawlers cannot. When ChatGPT, Perplexity, or Claude reach out to fetch a page for grounding or citation, the majority of that traffic comes from fetchers that pull raw HTML and do not run a full browser. If your content only exists after hydration, those engines see the empty shell, extract nothing worth quoting, and cite a competitor whose content was in the initial response. Headless done wrong now costs you rankings and citations at the same time.

Rendering strategies, compared

The choice that decides everything downstream is where and when your HTML gets assembled. There are four practical options, and each carries a different implication for indexing and citation. Client-side rendering (CSR) ships an empty shell and builds the page in the browser. Server-side rendering (SSR) runs the JavaScript on the server for each request and returns finished HTML. Static site generation (SSG) renders every page once at build time and serves pre-built files. Incremental static regeneration (ISR) is SSG that quietly re-renders individual pages in the background on a schedule or on demand, giving you static output that does not go stale.

The practical guidance is unambiguous for anything you want ranked or cited: get finished HTML into the first response. Static generation is the strongest default for marketing sites, documentation, blogs, and the comparison content that earns most AI citations, because the content is fully present, byte-for-byte identical for every visitor including every crawler, and served fast from a CDN. ISR extends that to catalogs of tens of thousands of pages that change too often to rebuild in full. SSR is the right call when a page genuinely differs per request, for example pricing that varies by region, and you still want crawlers to see complete markup. Reserve pure CSR for surfaces that are gated behind login and have no business being indexed at all.

A word of caution about hybrid frameworks. Next.js, Nuxt, Remix, Astro, and SvelteKit all let you mix these modes per route, which is powerful and dangerous in equal measure. We regularly find sites where the homepage and top landing pages are statically rendered and healthy, while an entire section, often the one added last, quietly defaulted to client-side rendering because a single data-fetching hook ran in the browser instead of on the server. The site passes a spot check and fails on the pages nobody re-tested. Rendering mode is a per-route property, so it has to be audited per route.

How crawlers and AI engines actually fetch you

Googlebot processes a JavaScript page in two waves. First it crawls the raw HTML and extracts whatever links and content are present immediately. Then the URL enters a render queue, where a headless Chromium instance executes the JavaScript and produces the final DOM, which Google indexes. The catch is the queue. Rendering is expensive, so it is deferred, and the delay between the first crawl and the render pass can run from hours to, on large or low-authority sites, weeks. During that window Google only knows what was in your raw HTML. If your content and internal links only appear after rendering, discovery and indexing of new pages slows to the pace of that second wave.

AI crawlers are a different and less forgiving population. GPTBot, PerplexityBot, ClaudeBot, and Google-Extended vary in sophistication, but the working assumption for 2026 should be that most AI fetching pulls raw HTML and does not execute a full browser render. Perplexity and ChatGPT lean heavily on live fetching to ground answers, and when that fetch returns an empty shell, there is no second render pass to save you. The engine extracts what it can from the raw response, finds nothing citable, and moves on. This is why a site can rank acceptably in Google, whose renderer eventually catches up, and remain completely absent from AI answers, whose fetchers never see past the shell.

The most important thing you can do is stop guessing about which population sees what. There is a definitive test, and it takes one command. Fetch your page with a tool that does not execute JavaScript, and look at the body. If the content, links, and metadata a crawler needs are present in that raw response, you are safe across every crawler and every AI engine. If they are missing, you have a rendering problem no amount of on-page copy will fix, because the copy is not in the document the fetcher receives.

# 1. What a non-rendering crawler (and most AI fetchers) sees
curl -sL https://example.com/pricing | grep -o '<h1[^>]*>.*</h1>'
#   empty result = your H1 is injected by JavaScript, not in the HTML
 
# 2. Count words in the RAW response
curl -sL https://example.com/pricing | sed 's/<[^>]*>//g' | wc -w
#   under ~50 words on a content page = client-side rendered shell
 
# 3. Compare against the RENDERED DOM (needs a headless browser)
# node render-check.js https://example.com/pricing
#   if rendered word count >> raw word count, crawlers are at risk
 
# 4. Confirm the AI user agents are allowed to fetch at all
curl -sL https://example.com/robots.txt | grep -iE 'GPTBot|PerplexityBot|ClaudeBot|Google-Extended'

Hydration, payload, and Core Web Vitals

Even when you server-render correctly, the browser still has to hydrate: download the JavaScript bundle and attach event handlers to the server-rendered markup so the page becomes interactive. Hydration is where a technically correct SSR site can still deliver a poor experience, because a heavy bundle blocks the main thread while it parses and executes. The content is visible, but the page is inert, and the metrics that gate rankings punish exactly this gap between visible and interactive. Getting the HTML right is necessary but not sufficient. The weight of the JavaScript that follows determines whether you pass Core Web Vitals.

Three field metrics decide the outcome. Largest Contentful Paint (LCP) measures when the biggest element in the viewport renders, and it should land under 2.5 seconds. On headless sites, LCP suffers when the hero content waits on a client-side API call that could have been resolved at build or on the server. Cumulative Layout Shift (CLS) measures visual stability, and it should stay under 0.1. It spikes when hydration injects content that pushes the layout around, or when web fonts and images arrive without reserved space. Interaction to Next Paint (INP) measures responsiveness to real input and should stay under 200 milliseconds. INP is the vital most headless sites quietly fail, because a large hydration bundle keeps the main thread busy and clicks feel laggy.

The fixes are structural, not cosmetic. Move the data the hero needs to build time or the server so LCP does not wait on a browser fetch. Split the JavaScript bundle and defer everything not needed for the first interaction, so hydration stops monopolizing the main thread and INP falls. Reserve explicit dimensions for images and embeds and preload the web font so CLS stops spiking. Measure all of this from field data, from real Chrome users, not from a single lab run on a fast laptop, because lab scores routinely look fine on templates that fail for the median visitor on a mid-range phone. Fix at the template level so a single change holds across every page built from it, rather than chasing one URL at a time.

Routing, canonicals, and duplicate content in SPAs

Single-page apps handle navigation in the browser without a full page load, which is fast for users and a minefield for crawlers if it is done carelessly. The non-negotiable rule is that every distinct piece of content must live at a real, server-addressable URL that returns the right content on a direct request. Two anti-patterns break this. The first is fragment routing, where the app encodes state after a hash, as in /products#gpu-42. Crawlers ignore everything after the hash, so every such variant collapses to one URL and only one page ever gets indexed. The second is a router that only works after client-side navigation, so a direct request to a deep URL returns the homepage or a soft 404. Both are fatal to indexing and both are easy to ship without noticing, because in-app navigation hides them.

Canonicals are the next trap, and headless makes them worse. If your canonical tag is written by a client-side library after hydration, a non-rendering crawler reads whatever placeholder was in the raw HTML, which is frequently the homepage URL or a blank. We have seen entire catalogs consolidate to a single canonical because a routing library set the tag late and the same default shipped in the static shell for every page. Canonicals, like titles and descriptions, must be present and correct in the server response for the specific URL, not patched in on the client. Verify them against raw HTML, not the rendered DOM your browser shows you.

Duplicate content is the quiet tax on top of all this. SPAs love query parameters for filters, sorting, and session state, and each unique parameter string is a distinct URL to a crawler. A category page with five filters and three sort orders can generate hundreds of URLs that all serve near-identical content, splitting ranking signals and burning crawl budget on pages that should never have been crawlable. Decide deliberately which parameters create genuinely distinct pages, canonicalize the rest to a clean base URL, and make sure your internal links point at canonical forms rather than parameter soup. On large sites this single discipline recovers more crawl budget than any other fix.

“In a single-page app, a URL that only resolves after a click is not a URL a crawler can index. If a direct request does not return the page, the page does not exist as far as search or AI is concerned.”

Structured data and llms.txt for headless

Structured data matters more on headless sites, not less, because it gives crawlers and AI engines an unambiguous, machine-readable statement of what a page is, independent of how the visible content was assembled. The critical requirement is that JSON-LD must be present in the server-rendered HTML. Schema injected by a client-side tag manager after hydration is invisible to non-rendering fetchers and unreliable even for Googlebot's deferred render pass. Emit it server-side, in the document body or head, for every page that ships. For a headless stack this usually means generating the JSON-LD in the same server or build step that produces the page, from the same CMS data, so the structured claims and the visible content can never drift apart.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Technical SEO for headless and JavaScript sites",
  "author": {
    "@type": "Person",
    "name": "Josh Bernstein",
    "jobTitle": "Managing Partner"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Something Inc."
  },
  "datePublished": "2026-05-01"
}
</script>
<!-- Must be in the RAW server response, not injected after hydration -->

Focus schema on the types engines actually parse and use. Organization, Article, author as a Person, Product, FAQPage, and BreadcrumbList do real work in both classic rich results and AI summaries. A long tail of rarely-parsed types adds markup weight without lifting citation rate, so deploy the useful ones, validate them, and drop the rest. Validation is not optional on headless, because a template-level error propagates to every page built from that template. One malformed field silently disqualifies thousands of pages from rich results at once, and you will not see it in a browser.

The newest access layer is llms.txt, a plain text file at your root that tells AI engines who you are and where your best, most citable content lives. For a headless site it is doubly valuable, because it hands engines a clean, curated map that does not depend on their ability to render your JavaScript to discover your key pages. Ship it as a static file, keep it current, and confirm in the same pass that your robots.txt is not blocking the AI user agents. A blocked GPTBot or PerplexityBot line is the most common reason a technically excellent site earns zero AI citations, and it is a one-line fix that teams overlook precisely because it is so small.

A diagnosis workflow and a prioritized fix plan

Diagnosis on a headless site follows one principle: always compare what the server sends against what the browser builds. Start with the raw-versus-rendered check from earlier, run across a representative sample of every template, not just the homepage. If raw word count, links, or metadata fall far short of the rendered DOM, you have found a rendering problem and you can stop looking for subtler causes. Next, pull server log files and separate the requests by user agent. Logs are ground truth: they show you which pages Googlebot, GPTBot, PerplexityBot, and ClaudeBot actually requested, how often, and what status they received. A template that never appears in the logs is a template that is never being discovered, usually because its links only exist in the rendered DOM.

Layer in Search Console coverage and the URL Inspection tool, which shows you the rendered HTML Google produced and flags pages excluded from the index with the reason. Cross-reference the excluded set against your log analysis and your raw-HTML sample. When those three sources agree, for example a template that is thin in raw HTML, rarely crawled in logs, and marked Discovered but not indexed in coverage, you have a confirmed, ranked problem rather than a hunch. This triangulation is what separates a real diagnosis from a guess, and it is the reason headless audits take longer than monolith audits: the truth is spread across three tools instead of sitting in one document.

Sequence the fixes by impact over effort, and do not let a hard problem block a cheap one. The two P0 items come first because they are the difference between being readable at all and being invisible: an errant robots.txt line is minutes of work with outsized payoff, and moving content into the server response is the foundation everything else sits on. The P1 metadata and URL fixes make the now-readable content indexable and correctly attributed. Only then does the P2 performance work, which raises how well you rank rather than whether you appear, earn its place. Ship llms.txt alongside, because it is nearly free and starts influencing AI citation within weeks. Work the list top to bottom, re-run the raw-versus-rendered check after each change, and you convert a headless site from a liability into an asset that Googlebot ranks and AI engines quote.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.