When Googlebot Spends Its Day on Pages You Never Wanted Indexed
A home goods retailer we reviewed had roughly 80,000 products. Their server logs told a different story: Googlebot was requesting more than 2 million distinct URLs a month, and fewer than 9 percent of those requests landed on actual product or category pages. The rest were color filters stacked on size filters stacked on sort orders, internal search results, and pagination that ran 40 pages deep. New seasonal products were taking three weeks to get indexed while Googlebot burned its allowance on URLs like ?color=blue&sort=price-asc&page=12.
This is the practical reality of crawl budget on large catalogs. The problem is rarely that Google refuses to crawl your site. The problem is that Google crawls the wrong parts of it, and your revenue pages wait in line behind millions of near-duplicate parameter URLs that should never have been crawlable in the first place.
What Crawl Budget Actually Is (And Who Needs to Care)
Google defines crawl budget as the set of URLs that Googlebot can and wants to crawl, governed by two forces. The first is the crawl capacity limit: the maximum number of parallel connections Google will use and the delay between fetches, which flexes based on how healthy and fast your server is. The second is crawl demand: how much Google wants your content, driven by perceived inventory, popularity, and how stale Google thinks your pages have become. You can read the full breakdown in Google’s official crawl budget documentation.
Crawl budget is not a concern for most websites. Google is explicit that this only becomes a meaningful constraint at scale, and the thresholds have not moved since 2020. Gary Illyes reconfirmed in 2025 that the rough lines are more than 1 million unique pages with content that changes weekly, or more than 10,000 pages that change daily. If you also see a large and growing pile of “Discovered, currently not indexed” URLs in Search Console, you are likely in crawl budget territory regardless of your raw page count.
The line between a real crawl budget problem and a content problem
It is worth being honest about this, because plenty of agencies sell crawl budget audits to sites that do not need one. For most business websites, what looks like a crawl budget issue is actually a content quality problem, a broken internal linking structure, or a slow server. Enterprise catalogs and large publishers are the genuine exception. They generate URLs faster than Google can keep up, and the math of faceted navigation is the main reason why. Diagnosing which of these you are facing is exactly the kind of work a proper technical SEO and GEO audit exists to settle before anyone touches a robots.txt file.
Where the Budget Leaks: Faceted Navigation, Parameters, and Thin Pages
The single largest source of waste on ecommerce catalogs is faceted navigation, the filter and sort controls that let shoppers narrow a category. The trouble is combinatorial. Take one category with 50 products, 5 color options, 4 size options, and 3 sort orders. That alone produces thousands of unique URL combinations for a single category. Multiply that across hundreds or thousands of categories and you have millions of crawlable URLs that all show slight variations of the same inventory.
Google is direct about why this hurts. In its guidance on faceted navigation, it explains that crawlers cannot tell whether a faceted URL is useful without crawling it first, so they typically access a very large number of these URLs before concluding the pages are useless. Google adds that crawling faceted URLs tends to cost sites large amounts of computing resources because of the sheer volume of URLs and the rendering work each one demands. Every request Googlebot spends confirming that ?color=blue is a duplicate is a request it did not spend on a new product page.
The same pattern repeats with other URL types that quietly multiply:
- Parameter URLs from tracking, session IDs, and affiliate codes, which create infinite variations of identical content.
- Internal search result pages, which generate an unbounded set of low-value URLs Google calls infinite spaces.
- Deep pagination, where page 30 of a listing offers almost no unique value but still consumes crawl requests.
- Thin and soft 404 pages, including empty filter combinations that return a 200 status instead of a 404, which Google specifically flags as wasted budget.
- Long redirect chains and orphan pages with no internal links pointing to them.
Industry log file studies commonly estimate that 30 to 50 percent of crawl budget on large sites is spent on these non-essential pages, though the real figure varies widely by site and architecture. The only way to know your number is to look at your own logs.
Why Log Files Are the Source of Truth
Search Console tells you what Google indexed. It does not tell you which URLs Googlebot actually requested, how often, or what status codes it received. Server logs are the only record of request-level crawler behavior, which is why log file analysis sits at the center of any serious technical SEO program for a large site.
A log file analysis answers questions no other tool can:
- What share of Googlebot’s requests hit revenue pages versus parameter and filter URLs?
- Which directories or parameters absorb the most crawl activity relative to the value they produce?
- How quickly does Googlebot discover and crawl newly published products?
- Are crawlers wasting requests on 404s, soft 404s, and redirect chains?
- How often does Googlebot return to your highest-margin categories?
Verify the bot before you trust the data
One practical caution: user-agent strings are trivially forged, so do not assume every line claiming to be Googlebot really is. Verify it. Run a reverse DNS lookup on the requesting IP, which for genuine Googlebot should resolve to googlebot.com, google.com, or googleusercontent.com, then run a forward lookup back to the same IP. Alternatively, check requests against Google’s published IP ranges. This matters more every year as AI crawlers proliferate. Log studies for 2026 found that GPTBot traffic grew roughly 305 percent year over year and that AI bots now average around 4 percent of all HTML requests, so your raw logs contain a lot of non-Google activity you need to separate out, as log file analysis research documents in detail.
The Fixes, in Priority Order
Once your logs show where the budget leaks, the remediation is methodical. The goal is simple: make the URLs you do not want indexed uncrawlable, and make the URLs you do want indexed fast and easy to reach.
1. Block crawl traps in robots.txt, not noindex
For faceted parameters and internal search results that you never want in the index, Google’s recommended fix is to disallow them in robots.txt. This is a critical distinction that trips up many teams: a noindex tag still requires Google to crawl the page to see the tag, so it does not save crawl budget. A robots.txt disallow stops the request before it happens. Allow crawling of individual product pages and a single clean, unfiltered listing page, and block the filtered permutations.
2. Canonicalize the parameter URLs you cannot block
Where filters must remain crawlable for legitimate reasons, every variant should carry a rel=canonical pointing to the clean base URL. Keep parameters in a consistent order, use the standard ampersand separator, and return a real 404 for empty or nonsensical filter combinations rather than a 200 status. Google notes that canonicalization reduces crawl volume over time, though it is slower and less absolute than a robots.txt block.
3. Eliminate soft 404s and redirect chains
Return proper 404 or 410 status codes for removed products and empty results, collapse multi-hop redirect chains to a single hop, and prune sitemaps so they list only canonical, indexable, 200-status URLs with accurate lastmod dates. Clean sitemaps are one of the few direct signals you can send Google about what deserves attention.
4. Fix server speed, because speed now buys budget
Google states there are only two ways to genuinely increase crawl budget: add server resources, and improve content quality and value. Both run through your infrastructure. If your server struggles to serve bytes, Google’s fetchers automatically back off to avoid overloading you, which directly reduces how often they crawl. In January 2026 Google also reduced the HTML fetch limit from 15MB to 2MB per URL, meaning anything past the first 2MB of an HTML document is not fetched, rendered, or indexed, as covered in Search Engine Land’s report on how crawling works in 2026. For bloated enterprise templates this is a real risk, so place titles, meta tags, canonicals, and structured data early in the HTML and push heavy CSS and JavaScript into external files.
5. Strengthen internal linking to the pages that earn
Crawl demand follows internal links and popularity. Flattening deep category trees, linking prominently to high-margin and seasonal pages, and trimming low-value internal links all push Googlebot toward the inventory that drives sales. This work compounds with conversion gains, which is why crawl optimization and ecommerce demand generation are most effective when planned together rather than in isolation.
What Good Looks Like After the Work
You will not measure success in vanity metrics. You measure it in the logs. A healthy outcome is a rising share of Googlebot requests landing on revenue pages, faster time-to-index for new products (days rather than weeks), a shrinking count of “Discovered, currently not indexed” URLs, and steadier recrawl frequency on your most important categories. On the catalog described at the top of this article, blocking faceted permutations and cleaning the sitemap moved revenue-page crawl share from under 9 percent to over 40 percent within two crawl cycles, and new product indexing dropped from roughly three weeks to under four days.
Crawl budget optimization is not glamorous, and it does not produce a screenshot you can frame. It is plumbing. But on a large catalog, plumbing is what determines whether your newest, highest-intent pages reach searchers this week or next month. If you suspect Googlebot is spending its time in the wrong places, the answer is in your server logs, and the fix is a disciplined sequence of blocks, canonicals, and speed work that puts your revenue pages back at the front of the queue.
Sources
See where you are cited today
A free snapshot audit of your rankings and AI citations before we ever talk.