Crawl Budget: Who Needs to Worry and What to Do
Crawl budget is the set of URLs Google can and wants to crawl on your site. Google says it matters mainly for very large or frequently updated sites, not for most. This guide explains crawl capacity and crawl demand, who is affected, the practices Google recommends, and what it does not cover.
- Who it applies to
- Capacity vs demand
- Best practices
- What is not stated
crawl budget = capacity limit (server health) + crawl demand (size, popularity, staleness)
Google describes crawl budget as an advanced topic for very large or frequently updated sites. Rough guideline thresholds: about 1 million+ unique pages changing weekly, 10,000+ pages changing daily, or many URLs marked Discovered – currently not indexed. These are rough estimates, not exact cutoffs. For most sites, a current sitemap and checking the Page indexing report is sufficient.
Who needs to worry
The guide is not needed if your site is small or your pages are crawled the same day they are published. Google also treats each unique hostname as a separate site with its own budget, so www and a subdomain have separate budgets.
Crawl capacity limit (hostload)
This is the amount of time your server spends holding connections open for Google, based on parallel connections and their duration. Every site starts with the same conservative default. The limit rises when the site stays healthy and falls when it slows down, returns 5xx errors or sends rate-limiting signals such as HTTP 429. It is shared across all Google crawlers, and Google’s own resources are also a factor. See server errors and TTFB.
Crawl demand
| Factor | Meaning per Google |
|---|---|
| Perceived inventory | Google tries to crawl every URL it knows; duplicates and unwanted URLs waste time |
| Popularity | Popular URLs are recrawled more often |
| Staleness | Google recrawls to catch changes |
Site-wide events such as site moves can temporarily increase demand. Even if capacity is not maxed out, low demand means less crawling.
Manage your URL inventory
Google also says not to use robots.txt to temporarily free budget for other pages: freed budget is not redirected unless Google is already hitting the capacity limit. See faceted navigation.
Make pages efficient
- Improve server response times and page resources so Google can read more content.
- Support 304 Not Modified so unchanged pages can be reused from cache, saving bandwidth.
- Check for availability problems during crawling.
Review crawl budget step by step
Check your site size
Compare with thresholds.
Open Page indexing
Look for discovered not indexed.
Review Crawl stats
Response times and errors.
Find wasted URLs
Duplicates, parameters.
Apply the practices
Consolidate, 404, sitemap.
Monitor
Compare after weeks.
See Crawl stats report and Page indexing report.
How to get more crawl budget
- Add server resources if capacity is the bottleneck, for example when URL Inspection reports Hostload exceeded.
- Improve content quality for the Google product you target; for Search, Google cites popularity, user value, uniqueness and serving capacity.
Notes for WordPress sites
- Tag and archive bloat: check which archives create thin duplicate URLs (archives guide).
- Parameters: filter and search URLs can multiply (my own note).
- Hosting: slow shared hosting can lower the capacity limit (my own inference); see speed up WordPress.
- Small sites: most WordPress blogs will not need this.
Troubleshooting
| Symptom | Likely cause (my own diagnosis) | Action |
|---|---|---|
| Many discovered not indexed | Large inventory or low demand | Consolidate and improve quality |
| Hostload exceeded | Server capacity | Add resources |
| Slow crawling | Slow responses or errors | Fix performance and 5xx |
| Crawling junk URLs | Parameters, duplicates | Block or consolidate |
Checklist
Applicability
- Size vs thresholds
- Discovered not indexed count
- Hostnames listed
- Small site? skip
Inventory
- Duplicates consolidated
- Junk URLs blocked
- 404/410 for removed
- Soft 404s fixed
Technical
- Fast responses
- No 5xx
- 304 supported
- Short redirects
Sitemaps
- Current
- Only crawlable URLs
- lastmod used
- Monitored
What Google’s page does not say
The page I used does not cover myths such as compression, nofollow and crawl-delay (it links to a separate article), any statement on crawl rate beyond the capacity mechanics, specific handling of 503 responses (it mentions only 5xx in general and 429), a detailed monitoring procedure, numeric thresholds for capacity or rates, or how long Google takes to adjust the limit.
Key terms
- Crawl budget
- URLs Google can and wants to crawl.
- Hostload
- Capacity limit for a host.
- Crawl demand
- How much Google wants to crawl.
- 304
- Not Modified response.
Common mistakes
Worrying on a small site
Usually unnecessary.
Using noindex to save budget
Google still requests the page.
Temporary robots.txt blocks
Budget is not redirected.
Leaving soft 404s
Keep being crawled.
Stale sitemaps
Poor signals.
Slow server
Lowers capacity.
Related guides
Crawl budget questions
Do I need to worry about crawl budget?
Usually only if your site is very large or frequently updated.
What are the rough thresholds?
About 1 million+ pages changing weekly, or 10,000+ changing daily, or many Discovered – currently not indexed URLs.
Is each subdomain separate?
Yes, each unique hostname has its own budget.
What lowers the capacity limit?
Slow responses, 5xx errors and rate-limiting signals like 429.
Does noindex save crawl budget?
No; Google still requests the page.
Should I block URLs in robots.txt?
For unimportant URLs, yes, but not as a temporary budget trick.
What response for removed pages?
404 or 410.
Does Google explain crawl-delay here?
Not on the page I used.
Definitions, thresholds, capacity and demand, best practices and ways to get more budget come from Google’s page. The step plan, WordPress notes, troubleshooting and checklists are my own.