What Is Crawl Budget and Why Does It Matter?
Crawl budget is how much attention Google gives to crawling your site. For most small sites it isn't a concern; for large or fast-changing sites, it can decide which pages get indexed.
Key takeaways
- Crawl budget is the number of URLs Google can and wants to crawl on your site in a given time.
- It combines crawl capacity (how much your server can handle) and crawl demand (how much Google wants to crawl).
- Google says it mostly matters for very large sites or sites that change rapidly.
- Wasted crawling comes from duplicate URLs, parameter explosions, redirect chains and low-value pages.
Crawl budget is the number of URLs Googlebot can and wants to crawl on your site over a period of time. If Google doesn’t crawl a page, it can’t index it, and new or updated content takes longer to appear in search.
The two parts of crawl budget
| Part | What it means | What affects it |
|---|---|---|
| Crawl capacity | How much crawling your server can handle without slowing down | Server speed and errors |
| Crawl demand | How much Google wants to crawl your site | Popularity, freshness and the number of URLs worth crawling |
See crawl budget vs crawl rate in the glossary.
Does crawl budget matter for your site?
Google’s guidance is that crawl budget mainly matters for:
- Very large sites (roughly a million+ unique pages)
- Medium or large sites with content that changes daily (for example 10,000+ pages)
- Sites with many “Discovered – currently not indexed” pages in Search Console
If your site has a few hundred or a few thousand pages and new pages get indexed quickly, crawl budget probably isn’t your problem. Focus on content quality instead.
How to check
- Search Console → Settings → Crawl stats: total crawl requests, average response time, responses by status and by file type
- Page indexing report: many URLs in “Discovered – currently not indexed” can mean Google isn’t getting to them
- Server logs (for large sites): exactly which URLs Googlebot requests
What wastes crawl budget
- Faceted navigation and URL parameters creating endless variations
- Duplicate content on multiple URLs
- Redirect chains and long lists of redirected URLs in internal links
- Soft 404s and error pages
- Low-value pages: thin tag archives, empty category pages, internal search results
- Slow server responses (which reduce crawl capacity)
How to make the most of it
- Block crawling of useless URLs (internal search, endless filters) in robots.txt.
- Consolidate duplicates with canonical tags and redirects.
- Fix redirect chains and broken links.
- Improve server speed and fix 5xx errors.
- Keep sitemaps clean with only canonical, indexable URLs. See XML sitemaps explained.
- Strengthen internal linking to important pages.
- Remove or noindex thin, low-value pages.
Crawl efficiency is covered in the technical SEO audit.