Why does crawl frequency drop on a section of the site?
- Published November 6, 2026
- Linked from 1 answer
- In a cluster of 2
Crawl frequency on a section drops for one of two broad reasons: Google’s ability to crawl fell, or its interest in that section fell. Ability is about your server, which Google crawls less when it slows down or returns errors. Interest is about the section itself: fewer links pointing into it, little change since Google’s last visits, or pages Google treats as duplicates. Check ability first, because it is the quicker one to confirm and to fix.
Start with the server side
Google’s crawl budget documentation says that if the site slows down or responds with server errors or rate-limiting signals, the crawl capacity limit goes down and Google crawls less. Its HTTP status code documentation adds that 5xx and 429 errors “prompt Google’s crawlers to temporarily slow down with crawling.”
A section can be hit harder than the rest of the site if it runs on a different backend, a slower database query, or a separate subdomain. Open the Crawl Stats report and compare response times and errors across the period of the drop, then check the section’s own server logs for 5xx responses or timeouts. If the section lives on its own hostname, remember that Google treats each hostname as a separate site for crawling.
Then look at demand for that section
| Possible cause | How to check |
|---|---|
| Internal links into the section were removed or moved deeper | Compare the navigation and hub pages before and after a redesign |
| The section stopped changing | Look at when its pages were last meaningfully updated |
| Pages are being treated as duplicates | Check the Page indexing report for duplicate and alternate statuses in that folder |
| The section grew a large number of parameter or filter URLs | Look at the URLs Googlebot requests in your logs |
| A robots.txt rule now covers part of it | Test sample URLs against the current robots.txt |
Google’s demand factors are the inventory of URLs it knows about, their popularity, and the staleness of its copies. A section that has lost links and stopped changing has weaker claims on all three.
What to do about it
Fix any server problem first, since a capacity drop lowers crawling for every page in the section regardless of quality. Then restore paths into the section from pages Google already crawls often, update pages when their content truly needs it, and clean up duplicate and parameter URLs so crawls land on real pages. At the level of single pages, the factors that decide how frequently Google returns are listed in why Google crawls some pages daily and others never. A drop that has no server cause and no change in links or content is not, by itself, a problem to fix: lower crawling of pages that have not changed can simply mean there was nothing new for Google to fetch.
Add a comment