Google crawl budget 2026: capacity is shared across crawlers

TL;DR · 60 SECONDSGoogle rewrote its crawl budget guidance on July 22, 2026 and moved it to a new crawling infrastructure hub outside Search Central. Three lines carry real weight: every site starts on the same conservative default capacity, that capacity is shared across all Google crawlers even though demand is per-crawler, and supporting 304 Not Modified lets Google reuse its cached copy instead of spending capacity on an unchanged page. On a large site those three facts turn crawl budget from a vague concern into an allocation problem you can actually manage.

Crawl budget has spent a decade as the topic enterprise SEOs argue about and nobody can act on. Google's July rewrite changes that, because for the first time the documentation says out loud how the ceiling is shared — and once you know what competes for it, you know what to cut.

The page moved as well as changed. Crawl budget guidance now lives at a dedicated Google crawling infrastructure hub rather than inside Search Central, with a last-updated stamp of 2026-07-22. Google framed the edit as a clarity and terminology pass, which is technically true and undersells it. Barry Schwartz flagged the rewrite at Search Engine Roundtable the same day. The terminology is now precise enough to plan against, and precision is the whole difference here.

What changed in the docs

Three statements are worth quoting nearly verbatim, because each one contradicts a piece of folk wisdom that is still circulating in enterprise SEO decks.

WHAT THE DOCS NOW SAY	THE FOLK WISDOM IT REPLACES	WHAT YOU DO DIFFERENTLY
Every site starts with the same default, conservative crawl capacity limit, which rises over time if demand exists and the site stays healthy	Big sites get big crawl budgets automatically because they are big	Treat capacity as earned through server health and demand, not granted by size. A slow origin caps you regardless of authority
Each crawler has a different crawl demand, but the crawl capacity limit is shared across all crawlers	Googlebot and the AI-related fetchers draw from separate pools, so grounding traffic is free	Model total Google fetch volume as one number. Grounding and indexing compete on large sites
Support 304 Not Modified: if a page has not changed since Google last crawled it, a 304 tells Google to reuse the cached version	Conditional requests are a nice-to-have for bandwidth	Make conditional GET support a release requirement. It converts wasted capacity into capacity for pages that changed

THE SENTENCE TO TAKE TO YOUR PLATFORM TEAMThe crawl capacity limit is shared across all Google crawlers. High demand from one crawler can reduce the capacity available to the others — which means AI grounding fetches can starve indexing on a site large enough to hit the ceiling.

Why a shared Google crawl budget changes the math

For most sites this is trivia. If Google can crawl everything you have in a day, allocation does not matter. Crawl budget becomes a live constraint at roughly the point where your URL count runs into the millions, your inventory churns daily, or your rendering is expensive enough that each fetch costs real origin time. Ecommerce catalogues, marketplaces, job boards, classifieds, large publishers and anything with faceted navigation all qualify.

For those sites, the shared-capacity line reframes the last two years of AI crawler debate. The conversation has been about whether to allow AI fetchers at all, on licensing and citation grounds. This is a different axis: even if you want them, they are drawing from the same tap as your indexing crawl. A surge in grounding requests against a slow origin does not just cost you bandwidth. It costs you index freshness on the pages that actually earn revenue.

Unchanged pages re-fetched with no conditional support46%

Parameter and faceted URL variants21%

Genuinely new or updated pages18%

Grounding and other non-indexing fetches15%

Illustrative allocation of one shared crawl capacity ceiling on a large catalogue site. A worked scenario to show the competition, not measured data.

Those proportions are ours and hypothetical, but the shape is the point. On most large sites we audit, the biggest single consumer of crawl capacity is not the AI fetchers everyone is arguing about. It is re-crawling pages that did not change, because the origin has no mechanism to say so. That is the item you can fix this quarter without a policy debate.

The 304 response is the cheapest lever you own

Conditional GET has been in HTTP since the 1990s and is still missing from a startling share of enterprise stacks, usually because a CDN or an application layer strips or ignores the headers that make it work. The mechanic is simple. Your origin sends a Last-Modified date or an ETag with the response. When Google comes back it sends that value in an If-Modified-Since or If-None-Match header. If nothing changed, you answer 304 with no body, and Google reuses what it already has.

What a working conditional exchange looks like● LIVE

# First fetch
GET /products/widget-9000 HTTP/1.1
 
HTTP/1.1 200 OK
ETag: "a1b2c3"
Last-Modified: Tue, 11 Aug 2026 09:14:00 GMT
Cache-Control: public, max-age=300
 
# Later fetch, page unchanged
GET /products/widget-9000 HTTP/1.1
If-None-Match: "a1b2c3"
If-Modified-Since: Tue, 11 Aug 2026 09:14:00 GMT
 
HTTP/1.1 304 Not Modified
ETag: "a1b2c3"
# no body, no render, capacity preserved

Three implementation traps account for nearly every failure we find. First, the ETag changes on every request because it is derived from a timestamp or a build hash rather than the content — so nothing is ever unmodified. Second, the CDN terminates the conditional request and serves a full 200 from its own cache without passing the validator through. Third, a personalization layer or an analytics injection mutates the HTML per request, making the page genuinely different every time even though nothing a reader would notice has changed.

Test it directly rather than trusting the config. Fetch a stable page, capture the ETag, fetch again with If-None-Match, and confirm you get a 304 and not a 200. Do it through the CDN, not against the origin, because the CDN is where it usually breaks. This is a twenty-minute check that most teams have never run.

Cutting demand you never wanted

The other half of the work is reducing the number of URLs asking to be crawled at all. Crawl demand is generated by your own site structure, and most large sites manufacture far more of it than they intend.

1Collapse parameter spaceFaceted navigation, sort orders, session identifiers and tracking parameters multiply a catalogue into combinations nobody will ever search for. Decide which facet combinations are genuinely indexable, and make the rest unlinkable rather than merely canonicalized — a canonical still costs a fetch.

2Fix soft 404s and redirect chainsEvery hop in a chain is a separate fetch against the shared ceiling, and every soft 404 is capacity spent to learn nothing. On a big site these are usually counted in the hundreds of thousands, and they are the least controversial thing on any technical backlog.

3Speed up the originThe docs are explicit that capacity rises when the site stays healthy and falls when it does not. Response time is the health signal you control most directly, and it compounds — a faster origin earns more capacity, which surfaces changes faster, which improves the demand signal.

4Keep sitemaps honestA sitemap full of URLs that redirect, 404 or carry stale lastmod values actively misdirects demand. Accurate lastmod values are one of the few direct inputs you have into which pages Google prioritizes, and inflating them to force re-crawls trains the system to ignore you.

5Decide the AI fetcher question deliberatelyGiven shared capacity, allowing every AI fetcher unlimited access on a very large site is a resource decision, not just a licensing one. Meter the ones you cannot justify, allow the ones that drive citations, and write the policy down so nobody makes the call from a network console under time pressure.

That last point is where this article meets a very expensive failure mode. Crawler access controls have moved to the edge, and we walked through what happens when that toggle is flipped without SEO in the room earlier today. Managing crawl capacity and blocking crawlers are different jobs with the same console, and conflating them is how sites disappear.

What the rewrite does not say

Documentation changes get over-read, so it is worth marking the boundaries of the claim before someone builds a strategy on top of a sentence that was not there. Four clarifications cover the misreadings we expect to see quoted back at us over the next month, and each of them saves a team from spending a quarter on the wrong work.

MYTH

Crawl frequency is not a ranking signalBeing crawled more often does not make a page rank better. Capacity work buys you faster discovery and fresher indexing of changes you have made. If the change was not worth making, crawling it sooner adds nothing.

SCOPE

Most sites still do not need thisIf your site has a few thousand URLs and a healthy origin, Google is already crawling everything it wants to. Spending a quarter on capacity engineering at that scale is effort taken from work that would actually move revenue.

NUANCE

Shared capacity is not a reason to block on reflexThe docs describe competition for a ceiling, not a penalty. Cutting off fetchers that drive citations to preserve capacity you are not actually exhausting trades a real benefit for an imagined one. Measure whether you are near the ceiling first.

NUANCE

A conservative default is not a punishmentEvery site starts in the same place, including sites with strong authority. Capacity is described as rising with demonstrated demand and sustained health, which makes it a slow-moving output of engineering quality rather than something to negotiate.

Instrumenting Google crawl budget by user agent

None of this is manageable without log data segmented by user agent. Search Console's crawl stats are a useful summary and they are not enough at this scale — you need raw server logs, joined to your URL inventory, reporting fetches, response codes and average response time per crawler per day.

304

the status code that converts a wasted fetch into a preserved one

1 ceiling

shared across every Google crawler hitting your origin

2026-07-22

last-updated stamp on the rewritten crawl budget documentation

Four numbers are worth putting on a dashboard and watching weekly: total Google fetches by agent, the share of those fetches answered 304, the share landing on parameter or non-canonical URLs, and median origin response time for crawler traffic specifically. When the shared ceiling starts to bind, those four move before rankings do. Our technical audits build this view first because everything downstream depends on it, and the same log data answers how AI crawlers differ from AI agents in what they actually request and why some crawlers never see your rendered content at all.

Do this next

Run the conditional request test through your CDN on three stable pages this week. If you get 200s where 304s belong, that is your highest-value ticket and it needs a platform owner, not an SEO one. In parallel, pull thirty days of server logs and split Google fetches by user agent, because you cannot argue about allocation until you can see it.

Then write the crawler access policy down — allowed, metered, refused — and keep it next to the ongoing SEO program rather than in a network runbook. Google's rewritten crawl budget documentation is short enough to read in one sitting, and it is now the reference to hand your platform team when they ask why 304 support is on the roadmap.

See where you are cited today

A free snapshot audit of your rankings and AI citations before we ever talk.

Schedule a free consultation

Tyler TruffiMANAGING PARTNER, SOMETHING INC.

Tyler leads work at the intersection of SEO and generative engines at Something Inc., helping B2B brands get ranked and cited across every major AI engine.

Google crawl budget: one capacity limit, shared by every crawler