How do you tell if crawl budget is your problem or just an excuse?
- Published November 6, 2026
- Linked from 1 answer
- In a cluster of 77
Check whether your important pages are being crawled. If they are crawled and indexed, crawl budget is not what is holding them back, unless updates take so long to be recrawled that the indexed version is out of date. If they are crawled but not indexed, the cause is not how often Google visits; it lies in what Google found or how it processed the page, from the selected canonical and the rendered HTML to the content itself. Crawl budget is a likely cause when valuable pages go uncrawled while crawling goes to low-value URLs, and Google names the sites where that is likeliest: very large or fast-changing sites, and sites with a large share of URLs discovered but not yet crawled. A site of any size can also be crawled less because its server is slow or returning errors; that is a capacity problem, and the Crawl Stats report shows it.
Three readings that settle it
| What you see | What it means |
|---|---|
| Important pages are indexed, but rank poorly | Not crawl budget, if the last crawl date in URL Inspection is recent. Look at relevance, quality, and competition |
| Important pages show Crawled – currently not indexed | Not crawl budget. Google fetched the page and has not indexed it; check the Google-selected canonical and the rendered HTML in URL Inspection, then the content |
| Important pages show Discovered – currently not indexed in large numbers, on a large site, while crawl requests go to parameter or filter URLs | Crawl budget is a plausible cause |
| Crawl Stats show rising response times, 5xx errors or 429 responses, on a site of any size | Google is crawling less to spare the server; fix capacity first |
Google’s crawl budget documentation describes the sites it applies to as very large sites with frequently changing content, and sites with “a large portion” of their URLs in the Discovered – currently not indexed status, and calls its size figures “a rough estimate” rather than exact thresholds. The same page says the crawl capacity limit goes down, and Google crawls less, when a site slows down or responds with server errors or rate-limiting signals such as HTTP 429. Those groups, with their size figures, are read in full in at what site size crawl budget actually starts to matter.
Where to read the evidence
- Page indexing report. Filter to your sitemap and see the status of the URLs you care about. The Page indexing help explains that Discovered – currently not indexed means “The page was found by Google, but not crawled yet”, and adds: “Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.”
- Crawl Stats report. Look at total requests, average response time, and host status. Rising response times or server errors point to a capacity problem on your side.
- Server logs. Check which URLs Googlebot actually requests. If most requests go to faceted or parameter URLs while key pages are rarely fetched, crawling is being spent on the wrong things.
What to do with each answer
If budget is ruled out, stop working on it and fix the page-level cause the reports point to. If the server is slow or erroring, fix capacity first, since Google crawls less when the site slows down. If crawling is spent on junk URLs, reduce the inventory: stop generating the URLs, block infinite parameter spaces in robots.txt, and consolidate duplicates. Each fix can be checked in the same reports once Google has recrawled the affected URLs, which is the only way to know whether budget was the problem in the first place.
Add a comment