At what site size does crawl budget actually start to matter?
- Published November 6, 2026
- Linked from 3 answers
- In a cluster of 77
Google names the sites its crawl budget guidance is written for, and size alone is not the whole test. Its crawl budget documentation lists three groups: “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”, “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”, and “Sites with a large portion of their total URLs classified by Search Console as Discovered – currently not indexed”. It adds that “These are not exact thresholds.” A site outside those groups can treat crawl budget as a low priority, though a server that is slow or errors still limits crawling on a site of any size.
The two parts of crawl budget
Google describes crawl budget as the set of URLs it “can and wants to crawl”, shaped by two elements:
| Element | What drives it |
|---|---|
| Crawl capacity limit | How fast and reliably your server responds; Google says that if the site slows down or returns server errors or rate-limiting signals, "the limit goes down and Google crawls less" |
| Crawl demand | How many URLs Google knows about on the site, how popular they are, and how stale its copies are |
Server health belongs to the capacity side, not the demand side. A fast, stable server lets Google crawl more at once; it does not make Google want more of your pages.
Why a small site can still look like a big one
The third group in Google’s list is the one smaller sites fall into. Faceted navigation, sort orders, session parameters, and calendar pages can create far more crawlable URLs than real pages. Google’s faceted navigation guidance warns that a parameter-based implementation “can generate infinite URL spaces”. When those URLs pile up, the site’s real pages compete with junk for attention, and the symptom shows up as a large share of URLs stuck as Discovered – currently not indexed.
How to check where you stand
- Count your real, indexable pages and compare that with the number of URLs Google knows about in the Page indexing report.
- Look at the Discovered – currently not indexed count. A large and growing share is the signal Google itself names.
- If the site has fewer than a thousand pages, Google’s Crawl Stats report help says you should not need to use that report or worry about that level of crawling detail.
The idea that blocking low-value pages frees budget for important ones is tested in whether blocking low-value pages frees up crawl budget. If you fall outside Google’s groups, an indexing problem probably points somewhere other than crawl budget, though URL Inspection and your server logs can confirm that Google is actually fetching the pages; how to tell if crawl budget is your problem or just an excuse separates the two. If you fall inside them, the first lever is the URL inventory: fewer junk URLs mean more of Google’s attention reaches the pages you care about.
Add a comment