What Is Crawl Budget Optimization?
Crawl budget optimization is the work of helping search engines spend their limited crawling time on your important pages instead of on duplicates, errors and low-value URLs. It matters mostly for large sites. Google says its guidance applies to roughly a million or more pages, or ten thousand or more that change daily. Smaller sites rarely need it.
| Site type | Does crawl budget matter? | What to do |
|---|---|---|
| Under a few thousand pages | No | Keep the sitemap current; check indexing |
| 10,000+ pages, changing daily | Yes | Monitor crawl stats; reduce waste |
| 1 million+ pages, changing weekly | Yes | Active crawl management |
| Any size, with many “Discovered – currently not indexed” URLs | Possibly | Investigate quality and server speed |
The thresholds are Google’s own, from its guide to managing crawl budget, which calls them rough estimates.
How Does Googlebot Decide What to Crawl?
Googlebot’s crawling is set by two things: how much it can crawl without straining your server, and how much it wants to crawl.
Crawl capacity limit
This is the maximum crawling Google will do without overloading your site. It rises when the site responds quickly and reliably. It falls when responses slow down or the server returns errors.
Crawl demand
This is how much Google wants to crawl. For Googlebot, it depends on the site’s size, how often content changes, page quality and popularity. Google says you influence demand most by managing your URL inventory: telling it which pages matter and removing duplicates.
Together, capacity and demand define the crawl budget: the set of URLs Google can and wants to crawl.
When Does Crawl Budget Really Matter?
It matters when a site has more URLs than Google is willing to crawl regularly, so important pages are found late or refreshed rarely. Signs include:
- New pages take weeks to be crawled, despite a sitemap and internal links.
- Many URLs sit under “Discovered – currently not indexed” in the Page indexing report.
- Server logs show Googlebot spending most of its requests on filtered, parameter or error URLs.
- Updated prices or stock take a long time to appear in results.
If new pages on your site are crawled within a day or so, Google says you do not need to worry. Publishing more useful content will help more than tuning crawl settings.
How Do You Read Crawl Stats?
Open the Crawl Stats report in Search Console, under Settings. It shows Google’s crawling of your site over the last 90 days.
| Section | What to look for |
|---|---|
| Total crawl requests | Trend over time; sudden drops or spikes |
| Average response time | Rising times reduce crawl capacity |
| Host status | Any availability problems |
| By response | Share of 200, 301, 404 and 5xx responses |
| By file type | Whether crawling goes to pages or to resources |
| By purpose | Discovery of new URLs versus refresh of known ones |
| By Googlebot type | Smartphone, desktop, image and others |
A healthy site shows mostly 200 responses, stable response times and no host problems. A large share of redirects, 404s or errors means crawl effort is being spent on dead ends.
For more detail, analyse server logs. They list every URL Googlebot requested, which the report only samples.
Want this handled for you?
Our SEO services start with a full audit and a written plan, followed by a monthly report that shows what changed and why.
See our SEO servicesWhich Fixes Save Crawl Time?
Google’s guide lists the fixes. In order of typical impact:
- Consolidate duplicate content. Use canonicals and consistent internal links, so crawling focuses on unique pages.
- Block crawling of URLs that should never be in Search with robots.txt, such as endless filter combinations and internal search results. See our guide to robots.txt optimization.
- Return 404 or 410 for permanently removed pages. A real error code tells Google to stop requesting the URL.
- Eliminate soft 404s. Pages that say “not found” while returning success keep being crawled.
- Keep sitemaps up to date, with accurate lastmod values. See our XML sitemap guide.
- Avoid long redirect chains.
- Make pages load efficiently. Faster responses let Google fetch more in the same time.
- Monitor availability, and fix server errors.
For stores, faceted navigation is the usual source of waste. Google’s guide to faceted navigation explains that crawlers may request a very large number of filter URLs before recognising they are not useful.
What Does Not Help?
Several popular ideas do not work.
- Noindex does not save crawl budget. Google must still crawl a page to see the tag. Use robots.txt to stop crawling, and noindex to stop indexing.
- The crawl-delay rule in robots.txt is not processed by Googlebot.
- Compressing sitemaps does not change crawl budget.
- Nofollow on internal links is unreliable for this purpose; the URL may be discovered elsewhere.
- Blocking and unblocking pages to “shift” budget toward other sections is something Google advises against.
- Returning 4xx codes to slow crawling, other than 429, is also discouraged.
Google’s guide also notes that crawling is not a ranking factor. More crawling does not mean higher positions.
How Do You Increase Crawl Capacity?
You raise capacity by making the server faster and more reliable. Google increases crawling by itself when a site responds well.
- Improve server response time.
- Fix intermittent 5xx errors.
- Use caching and a content delivery network.
- Keep page resources light.
You cannot request a higher crawl rate. Demand grows when the content is valuable and updated, and when other sites link to it.
What Should Small Sites Do Instead?
Small sites should ignore crawl budget and check three basics.
- A current sitemap is submitted.
- Important pages are linked from other pages.
- The Page indexing report shows no unexpected exclusions.
If a small site has indexing problems, the cause is almost always quality, duplication or a technical block, not budget. Our guide to the Google index covers those causes, and enterprise SEO covers crawl management at scale.
Frequently Asked Questions
Do small websites need to worry about crawl budget?
No. Google says sites without large numbers of rapidly changing pages, or whose new pages are crawled the same day they are published, do not need to read its crawl budget guide. Keeping the sitemap current and checking the indexing report regularly is enough.
How do I see how often Google crawls my site?
Open the Crawl Stats report in Search Console, found under Settings. It shows total crawl requests, average response time and a breakdown by response code, file type and purpose for the last 90 days. Server log files give the complete, URL-by-URL record.
Do redirects waste crawl budget?
Each redirect needs an extra request, so long chains do waste crawling and slow discovery. A single redirect from an old URL to its replacement is normal and fine. Update internal links to point at final addresses, and remove chains where one redirect leads to another.
Does noindex save crawl budget?
No. Google has to crawl a page to see its noindex tag, so the request is still made. To stop crawling, disallow the URL pattern in robots.txt. Do not combine the two on one URL, since a blocked page’s noindex cannot be read.
Does a faster site get crawled more?
Often, yes. Google raises its crawl capacity limit when a site responds quickly and reliably, and lowers it when responses slow or errors appear. Speed increases how much Google can crawl. Whether it wants to crawl more still depends on the value and freshness of your content.
Related Guides
Want this done for you?
See what each monthly plan includes, or send us your site and questions on WhatsApp. We do not promise rankings; we show the work and the data every month.
View SEO packages Chat on WhatsAppSources
