Something Inc.LoginSchedule a free consultation
TECHNICAL SEO

Technical SEO Services: Scoping For Two Crawlers

Googlebot renders JavaScript. The crawlers behind ChatGPT, Claude and Perplexity do not. Most technical SEO scopes are still written for one crawler population, and the gap is where citations quietly go missing.

TECHNICAL SEOAI CRAWLERSNOV 2026

Googlebot executes JavaScript. The crawlers that feed ChatGPT, Claude and Perplexity do not. Every technical SEO decision you make in 2026 sits downstream of that one asymmetry, and most scopes of work are still written as though there were a single crawler to satisfy.

This is not a small difference in degree. It is a difference in kind. A page that renders its main content client-side is a complete page to Google and a near-empty one to an AI crawler, which means the same URL can rank perfectly well and be structurally incapable of being quoted. Teams find this out late, usually after a quarter of content investment produces rankings and no citations.

So the scope question matters more than the vendor question. Below is what a technical engagement covers now, which findings rest on settled engineering and which rest on one two-year-old measurement, and where the money is actually worth spending.

TL;DR · 60 SECONDSThe split. Crawl, index, render, status codes, architecture and internal linking are mature engineering: cheap to diagnose, measurable in Search Console, and not in dispute. Machine access for AI crawlers is not mature, and the evidence base is thinner than vendors imply. The load-bearing measurement is Vercel's crawler analysis with MERJ, published December 17, 2024, which tested GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and CCBot and found none of them render JavaScript. Gemini and Applebot do, because they ride Googlebot and a browser-based crawler respectively. The same study found ChatGPT spending 34.82 percent of its fetches on 404s and Claude 34.16 percent, against 8.22 percent for Googlebot. Our verdict: buy the settled half as engineering with a fixed scope, buy the contested half as instrumented experiments with named methodology, and refuse any proposal that prices a citation lift off technical work alone.
34.82%
of ChatGPT crawler fetches landed on 404 pages in Vercel and MERJ's December 2024 analysis, against 8.22 percent for Googlebot
0 of 8
AI crawlers tested in that study executed JavaScript, including GPTBot, ClaudeBot, PerplexityBot and Bytespider
4% to 7%
of AI citations explained by standard SEO ranking factors, per Profound's February 2026 research
80%+
of Google searches now carry an AI Overview, per Search Engine Roundtable's November 2026 report

What Do Technical SEO Services Actually Include In 2026?

Technical SEO services are an engineering engagement that makes a site retrievable, parseable and quotable by machines: crawl access, indexing, server-side rendering, status code and redirect hygiene, site architecture, internal linking, structured data, page performance and, since 2025, explicit access rules for AI crawlers. The deliverable is merged code, not a report.

DEFINITIONTechnical SEO is the practice of removing every machine-side obstacle between a published page and the systems that index, retrieve and quote it, measured by whether those systems can reach the content, read it without executing code, and resolve the URL they hold for it.

Five workstreams make up a credible scope. They are listed here in the order a competent team sequences them, because each one makes the next measurable. Running structured data work before you have fixed rendering is how agencies produce a lot of activity and no movement.

WORKSTREAMWHAT IT FIXESWHO OWNS THE FIXMEASURABLE IN
Crawl access and budgetrobots.txt rules, AI crawler directives, crawl traps, faceted URL explosion, server response under loadAgency specifies, platform or DevOps shipsServer logs, Crawl stats report
Rendering and content deliveryMain content present in server-delivered HTML rather than injected client-sideFront-end engineering, with the agency defining the acceptance testView source, fetch without JS enabled
Status codes and URL stabilityRedirect chains, soft 404s, stale URLs still being requested by AI crawlers months after a migrationAgency audits, platform shipsServer logs, Page indexing report
Architecture and internal linkingOrphan pages, depth, hub and spoke coverage, anchor consistency across templatesShared: agency maps it, content and engineering implementCrawl depth distribution, internal link counts
Structured data and machine readabilitySchema that matches on-page facts, parseable tables, stable headings, no gated proofAgency specifies, engineering shipsRich Results Test, retrieval spot checks

Notice what is missing from that table. There is no row for an AI visibility score, because no technical change has a defensible conversion rate into citations. We argued the general version of this when we called retrieval access a gate rather than a dial, and it applies here with force: fixing rendering does not buy you citations, it buys you eligibility for them.

Why Technical SEO Services Now Scope For Two Crawler Types

Two crawler populations now read your site with materially different capabilities, and a scope written for Googlebot alone will pass its own tests while leaving the AI surface broken. Googlebot renders, retries, and reports. The AI crawlers mostly do none of the three.

The measurement everyone cites for this is Vercel's crawler analysis with MERJ, published on December 17, 2024, built on traffic to nextjs.org and the Vercel network plus two job board sites as a consistency check. It is worth being precise about what it found, because the secondhand versions overstate it in one direction and underrate it in another.

CRAWLEREXECUTES JAVASCRIPT404 SHARE OF FETCHESMONTHLY FETCHES OBSERVEDSCOPE IMPLICATION
GooglebotYes8.22%4.5 billionClient-side content is readable, URL inventory is current
GPTBot and ChatGPT crawlersNo34.82%569 millionNeeds server-rendered HTML and aggressive redirect hygiene
ClaudeBotNo34.16%370 millionSame, and it requested JavaScript files for 23.84 percent of fetches without running them
PerplexityBotNoNot broken outNot broken outServer-rendered HTML only
GeminiYesNot broken outNot broken outRides Googlebot infrastructure, so Google-side fixes carry over
ApplebotYesNot broken outNot broken outBrowser-based crawler, renders like a search engine

The 404 column is the finding buyers underweight, and it is the cheapest thing on this page to act on. A third of what the ChatGPT and Claude crawlers fetched was a dead URL, four times Googlebot's rate. Those crawlers are working from stale inventories assembled at training and retrieval time, and they do not re-discover your sitemap on your schedule.

That turns two unglamorous jobs into high-yield work. Keep redirects resolving in one hop and keep them in place for years rather than months, because the requesting population has a much longer memory of your old URLs than Google does. We went through the mechanics of this in detail when we looked at why AI crawlers cannot see client-rendered pages.

“Googlebot tells you when it cannot read your page. The AI crawlers fail silently, and the first signal you get is a competitor being quoted instead of you.”

Be fair about the age of this evidence, because it cuts both ways. The Vercel figures are from December 2024, which in this market is a long time, and no comparable large-scale re-test has been published since. Treat the direction as reliable and the exact percentages as dated. Then do the thing almost nobody does and verify it on your own property: request your top twenty pages without executing JavaScript and read what comes back.

Which Findings Are Settled Engineering And Which Are Contested?

A technical finding is settled when a named, reproducible mechanism connects the fix to the outcome, and contested when the connection rests on correlation across datasets that disagree with each other. Both belong in a scope. They do not belong at the same confidence, and they should not be priced the same way.

The contested half is contested for a reason you can check yourself. Published estimates of how many AI citations also rank in Google's top 10 run from 12 percent to 84 percent across five credible research teams, a spread we worked through when writing about what SEO audit services can honestly measure. Profound's February 2026 research put the share of AI citations explained by standard SEO ranking factors at 4 to 7 percent.

Status codes, redirects, URL stability95%
Server-side rendering of main content90%
Crawl access and robots directives85%
Architecture and internal linking70%
Structured data as a citation lever35%
Composite AI visibility scores10%

Confidence a buyer should assign to each technical workstream, scored on whether a named reproducible mechanism links the fix to the result. Illustrative scoring, based on the evidence quality discussed above

Read the bottom two bars as a buying rule. Structured data is genuinely useful for machine readability and genuinely oversold as a citation mechanism. A composite AI visibility score, computed by a vendor from a method it will not publish, is not a measurement you can act on or audit, and a proposal that leads with one is telling you how it intends to report success.

The top three bars are where a fixed-scope engagement earns its fee. They are diagnosable in days, the fixes are unambiguous, and the result is verifiable by someone other than the agency that did the work. That verifiability is the whole point, and it is the standard we hold our own SEO and GEO audits to before anything gets prioritised.

How Much Of This Should Your Own Engineers Keep?

Most of the implementation should stay in house, and the agency should be buying you specification, prioritisation and acceptance tests rather than commits. The exception is diagnosis, which is where outside pattern recognition is worth real money because your team sees one site and a good technical team sees fifty.

1Keep the shipping, buy the specificationYour engineers already own the deploy pipeline, the CI checks and the risk of breaking production. What they usually lack is a precise statement of the acceptance test: content present in server-delivered HTML, one-hop redirects, canonical matching, no soft 404s. Buy the test, ship it yourself.
2Buy log file analysis outrightServer logs are the only record of what every crawler actually requested, and nothing in Search Console or analytics shows AI crawler behaviour. Parsing months of logs across bot populations is specialist work with poor returns on learning it once. This is the clearest line item to outsource.
3Keep rendering decisions, demand the evidenceWhether a template moves to server-side rendering is an architecture decision with cost and latency consequences your team owns. What an agency owes you is the per-template evidence of what is missing without JavaScript, page by page, not a blanket recommendation to render everything.
4Never outsource URL governanceRedirect maps and URL stability policy outlive agency relationships, and the 404 rates above show the requesting population remembers old URLs for a very long time. Own the registry of what redirects where and why, and review it at every migration.
5Treat retrieval checks as a standing experimentAsk the engines directly and on a schedule: query the surfaces you care about for the topics you have published, record whether you are quoted and what is quoted instead. It is cheap, it is first-party, and it is the only evidence in this whole category you own outright.

The split above also tells you what a bad proposal looks like. If the bulk of the hours sit in implementation your team could do, and the diagnosis is a tool export with the agency logo on the cover, you are paying consultancy rates for throughput. The reverse shape, heavy on diagnosis and specification with your engineers shipping, is where the economics work.

How To Buy Technical SEO Services Without Buying A Crawl Report

Buying well comes down to five questions asked before signature, each of which a weak vendor answers with a tool name and a strong one answers with a method. Ask them in this order, because the first answer tells you how seriously to take the rest.

1AT THE PITCHAsk what they will fetch without JavaScript, and show me the output
THE MOVES
Request a live demonstration on three of your own URLs during the pitch, not a slide
Ask which templates they expect to fail and why, before they run it
Require the per-template result as a deliverable, not a summary grade
DONE WHENYou have a page-level list of what AI crawlers can and cannot read on your site, produced before any contract starts.
2SCOPING CALLAsk for the log file methodology in writing
THE MOVES
Which bot user agents they verify by reverse DNS rather than trusting the string
What window of logs they need and in what format
Whether crawl waste is reported by template and directory or only in total
DONE WHENThe methodology is specific enough that a second vendor could reproduce the numbers.
3IN THE PROPOSALAsk them to separate settled findings from contested ones
THE MOVES
Require every finding to carry its evidence type: reproducible mechanism, correlation, or vendor assertion
Ask which findings they would stake a guarantee on and which they would not
Reject any single composite score that blends the two
DONE WHENYour remediation backlog is sorted by evidence quality as well as by effort.
4BEFORE SIGNATUREPrice the remediation, not just the diagnosis
THE MOVES
Get an engineering estimate for the top ten fixes before signing the audit
Identify which fixes need a platform change versus a template change
Agree who ships what, with dates
DONE WHENYou know the full cost of acting on the report, which is usually several times the report.
5IN THE CONTRACTDefine the acceptance test for every fix
THE MOVES
Write the pass condition as something a third party can check
Add the checks to CI so regressions surface on deploy rather than next quarter
Re-run the no-JavaScript fetch after each template change
DONE WHENTechnical health is enforced by your pipeline instead of re-audited every year.

One caution about sequencing, because it is the most common expensive mistake. Teams arriving from a content programme want the structured data and the entity work first, since that is the part that sounds like it produces citations. If rendering and status codes are broken underneath, that work lands on pages the AI crawlers cannot read, and you will conclude that structured data does not work when what actually happened is that nothing you shipped was ever retrieved. We built a dependency framework for this sequencing for exactly that reason.

It is also worth saying what has not changed, because the 2026 framing can make teams abandon things that still pay. Google is still the larger source of qualified traffic for almost every B2B site we work on, the November 2026 core update is rolling out right now, and Search Console still reports impressions without clicks on its AI surfaces. Classical technical SEO did not stop mattering when a second crawler population arrived. It got a second audience with stricter requirements.

DO THIS NEXTPick your ten highest-value pages and request each one without executing JavaScript. If the main content is missing, you have found your first quarter of technical work and you found it in an afternoon. Then pull one month of server logs and count what share of AI crawler requests returned a 404. If it is anywhere near the third that Vercel measured, redirect hygiene is the highest-return fix available to you and it is engineering you already know how to do. If you would rather hand the diagnosis and the per-template specification to someone who has run it across dozens of sites, that is what our technical SEO services do inside a broader search engine optimization engagement, and the no-JavaScript fetch is the first thing we will show you.

Common Questions About Technical SEO Services

The questions below are the ones that come up in every scoping call, answered short enough to forward to an engineering lead who has no interest in search terminology.

What are technical SEO services?An engineering engagement covering crawl access, indexing, rendering, status codes, site architecture, internal linking, structured data and page performance, plus explicit access rules for AI crawlers. The output is specified, prioritised fixes that get merged, not a scored report.
Do AI crawlers execute JavaScript?Mostly no. Vercel and MERJ's December 2024 analysis found GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and CCBot all failing to execute JavaScript. Gemini and Applebot do render, because they use Googlebot and a browser-based crawler.
How do I test whether my pages are readable by AI crawlers?Request the page with JavaScript execution disabled and read the returned HTML. If the main content, headings and links are absent, an AI crawler sees the same emptiness. Do this per template rather than per page, since templates fail as a class.
Why do AI crawlers hit so many 404 pages?They work from URL inventories assembled at training and retrieval time, not from your current sitemap. Vercel measured 34.82 percent of ChatGPT fetches and 34.16 percent of Claude fetches landing on 404s, against 8.22 percent for Googlebot.
How long should I keep redirects in place?Years, not months. The crawler populations requesting your old URLs refresh their inventories on their own schedule, so a redirect retired after six months turns previously quotable content into a dead end that nothing tells you about.
Does structured data get you cited by AI engines?It helps machines parse you and it is not a citation lever on its own. Profound's February 2026 research put the share of AI citations explained by standard SEO ranking factors at 4 to 7 percent, so treat schema as readability work rather than a growth tactic.
What does technical SEO cost?The diagnosis is rarely the expensive part. Price the remediation before you sign: get an engineering estimate for the top ten fixes, split by template change versus platform change, because that total is usually several times the cost of the audit itself.
Should technical SEO work stay in house?Implementation should, since your engineers own the pipeline and the production risk. Buy diagnosis, specification and log file analysis, which reward pattern recognition across many sites, and keep the shipping and the URL governance.
Is a composite AI visibility score worth paying for?No. A score computed by a method the vendor will not publish cannot be audited or acted on, and it blends settled engineering findings with contested ones. Ask for findings tagged by evidence type instead.
Does classical technical SEO still matter in 2026?Yes. More than 80 percent of Google searches carry an AI Overview and a core update is rolling out this month, so crawl, index and ranking health still drive most qualified traffic. The AI crawlers added an audience with stricter requirements, they did not replace the old one.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.

JB
Josh BernsteinMANAGING PARTNER, SOMETHING INC.

Josh leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.

Keep reading

All insights →
TECHNICAL SEOSEO Audit Services: What To Buy In 2026Five credible studies answered the same question about AI citations and rankings and came back with 12%, 17%, 38%, 76% and 84%. That spread is the most useful finding in search right now, and it decides what a 2026 SEO audit has to measure, what it has to stop pretending to know, and what the remediation actually costs.TYLER TRUFFI · 13 MINTECHNICAL SEOThe AI Training Opt Out Is Not The One You WantOn October 30, 2026, Google said it plans to honour a shared AI training opt out in robots.txt from 2027, and Microsoft published a full replacement for NoArchive and NoCache that splits AI Training from AI Grounding. The two announcements are being read as one thing: a line you add to robots.txt to opt out of AI. That reading is wrong in both directions. The shared token covers model training, which was never what cost publishers traffic. The grounding preference, which decides whether your pages feed an AI answer, cannot be set in robots.txt at all on Bing, and Google's generative search opt out remains a separate control in Search Console. Bing is blunt about the price of the grounding lever: pulling it will significantly reduce brand visibility in AI experiences. Here is what each of the four controls actually does, the deadlines that now sit in the calendar, and the setting almost every commercial site should land on.JOSH BERNSTEIN · 11 MINTECHNICAL SEOGoogle Expanded User Generated Spam Rules To App BuildersOn October 26, 2026 Google updated its Prevent user-generated spam on your site and platform guide, widening the scope line to name hosting platforms and tools that let users publish shared pages, generated content, or web apps, and adding a section titled Only index content meant for Search. That section tells platforms to decide whether a user-created page exists for public discovery or only for whoever holds the link, and to apply a noindex rule or block crawling by default when it is the second. A monitoring section tells platforms to verify in Search Console, watch Page Indexing and Search performance for unexpected changes, and track platform-level creation and indexing metrics with alerts on anomalies. Eighteen days earlier Google had documented the UGC Fresh Data Program, which exists to get user-generated content indexed faster. The two documents are not in conflict, but they do hand product teams a sorting job that almost nobody owns, and the wrong tool for that job quietly makes the problem permanent.TYLER TRUFFI · 12 MIN
Free consultation

Let us be the last SEO agency you ever work with

A 30 minute call and a free audit of your SEO and GEO position. You keep the findings either way.