What is crawl budget in SEO?

Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. It is determined by crawl rate limit (how fast your server can respond) and crawl demand (how popular and fresh your pages are). Large sites with thousands of URLs need to manage crawl budget actively; smaller sites under 1,000 pages rarely need to worry about it.

How do I check my crawl budget?

Use Google Search Console's Crawl Stats report (Settings > Crawl stats) to see Googlebot's request volume, response codes, and file types crawled over the past 90 days. You can also download a server log file and filter for Googlebot user agent strings to see exactly which URLs are being crawled and how often.

Does crawl budget matter for small sites?

Google's own documentation says crawl budget is not something most sites need to worry about if they have fewer than 1,000 URLs. It becomes a real concern once your site grows past that, especially if you have faceted navigation, URL parameters, session IDs, or a large number of thin or duplicate pages consuming Googlebot's attention.

Crawl Budget: How to Optimise It for SEO

What Is Crawl Budget?

Crawl budget is not a single fixed number Google assigns you. It is the outcome of two interacting factors: crawl rate limit and crawl demand. Google's John Mueller has explained this publicly more than once, and yet it still gets misrepresented as a simple quota.

Crawl rate limit is about your server. Googlebot backs off if your server is slow or returns errors, so a site with fast response times and high availability effectively gets a higher crawl rate limit. Crawl demand is about your content. Pages that get external links, get traffic, or are frequently updated attract more Googlebot attention than pages that have been sitting untouched for two years.

The two factors multiply. A fast server filled with high-demand pages gets crawled heavily. A slow server with stale, thinly linked pages gets crawled infrequently. Most sites fall somewhere in between, which is why crawl budget optimisation is usually about removing the drag rather than amplifying the signal.

Why Does Crawl Budget Matter for SEO?

New pages cannot rank until they are crawled and indexed. If Googlebot is spending half its crawl capacity on paginated archive pages, session ID variants, and print-friendly URL duplicates, your freshly published content is queuing behind that noise.

I have seen this concretely on e-commerce sites with faceted navigation. A product catalogue with 5,000 products but 200,000 URLs from colour/size filter combinations can easily starve Googlebot of the budget it needs to crawl the actual product pages. When we blocked the filter URLs in robots.txt and consolidated canonicals, crawl coverage of the real product pages improved measurably within 30 days, as visible in Search Console Crawl Stats.

For content-heavy editorial sites, crawl budget problems tend to surface differently. Infinite scroll pagination, tag and category pages that aggregate content without adding value, and URL parameters from analytics or session tracking are the usual culprits.

How Do You Diagnose a Crawl Budget Problem?

Start with Google Search Console. Go to Settings, then Crawl Stats. You will see total Googlebot requests per day over the last 90 days, broken down by response code. What you want to look at specifically is the breakdown by page type and by response code.

Real benchmark from our Crawl Stats tool: Sites with healthy crawl efficiency typically see 80%+ of Googlebot requests returning 200 status codes, with redirect chains under 5% of total requests. If you are seeing 20%+ 3xx or 4xx responses, that is crawl budget being burned on redirects and dead ends.

The second diagnostic is server log analysis. Download your server logs, filter for the Googlebot user agent (Googlebot, and separately Googlebot-Image, Googlebot-Video), and ask two questions: which URLs are being crawled most often, and which URLs that you care about are being crawled least often. The mismatch between "what Google is crawling" and "what you want Google to crawl" is the crawl budget problem statement.

Tools like Screaming Frog can simulate a crawl from your sitemap and flag URL parameter duplication, thin pages, and redirect chains that you can then cross-reference against your log data.

What Wastes Crawl Budget?

There are predictable categories. Each one is fixable.

Crawl Waste Source	Why It Happens	Fix
URL parameters	Tracking params, sort orders, session IDs create duplicate URLs	Block in robots.txt or consolidate with canonical
Faceted navigation	Filter combinations multiply URL count exponentially	noindex or block low-value facets; canonical to base URL
Redirect chains	Old URLs redirect to new URLs that redirect again	Update all links to point to final destination; collapse chains
Soft 404 pages	Empty search result pages, out-of-stock products returning 200	Return real 404 or 410; or consolidate and canonical
Paginated archives	Deep pagination pages with little unique content	noindex deep pages; use rel=next/prev signals where applicable
Duplicate content variants	www vs non-www, HTTP vs HTTPS, trailing slash variants	Enforce single canonical variant with 301s

How Do You Optimise Crawl Budget?

There are three levers: reduce waste, improve server speed, and increase crawl demand. In practice, reducing waste is usually where you get the fastest wins.

Block Low-Value URLs in Robots.txt

Use the Disallow directive in robots.txt to prevent Googlebot from crawling URL patterns that generate waste. The most common targets are internal search result pages (/search?q=), print-friendly versions (?print=true), and session ID parameters. Note: blocking in robots.txt does not de-index the URL if it already has links pointing to it. Use noindex for that.

Check your robots.txt implementation with the Robots.txt Tester before pushing changes, so you do not accidentally block Googlebot from your important pages. This is embarrassingly common.

Audit and Fix Your XML Sitemap

Your XML sitemap should only list canonicalised, indexable URLs that return 200. Including redirects, noindex pages, or soft 404s in your sitemap sends mixed signals and wastes crawl budget on pages you do not actually want indexed. The sitemap is essentially your content wishlist for Googlebot; keep it honest.

Improve Server Response Times

A server that regularly responds in over 500ms depresses your crawl rate limit. Google does not want to hammer slow servers. Use Google's website speed test tool to check your Time to First Byte (TTFB). Under 200ms is the target; over 500ms is actively hurting you. CDN caching, server-side rendering for static content, and database query optimisation are the usual solutions.

Increase Internal Linking to Important Pages

Crawl demand is partly shaped by how well linked a page is internally. Orphan pages, which have no internal links pointing to them, get crawled infrequently. A structured internal linking strategy, especially from your high-traffic hub pages to your newer content, signals to Googlebot that those URLs matter. See the full approach in the technical SEO audit guide.

Is Crawl Budget Only for Large Sites?

Google has said explicitly that sites with fewer than 1,000 URLs do not need to think about crawl budget. And that is broadly true. But I would add a nuance: it is not about page count alone, it is about the ratio of crawlable URLs to pages you actually want indexed.

A 500-page blog with clean architecture and no parameter bloat is fine. A 500-page e-commerce site with faceted navigation generating 50,000 URL variants is not, even though it has only 500 real product pages. So the better rule is: if your crawlable URL count is more than 5x your indexable page count, you have a crawl budget problem worth investigating.

Crawl Budget and AI Search Crawlers

This is a newer dimension that is not covered in most crawl budget guides. Google's AI Overviews, Bing Copilot, Perplexity, and other AI search tools use their own crawlers. Google's AIO bot, GPTBot for ChatGPT, and PerplexityBot all make independent crawl requests. They each have their own crawl budget relative to your server.

If you want your content to appear in AI-generated answers, you need to ensure these crawlers can access your pages. Check your robots.txt to confirm you are not accidentally blocking them. GPTBot, for instance, respects the User-agent: GPTBot directive in robots.txt. A blanket Disallow in your robots.txt can block your content from AI citations without you realising it.

Track which AI crawlers are hitting your site in your server logs. Cross-reference that against the AI citations guide to understand how those crawls translate to visibility in AI answers.

How Do You Monitor Crawl Health Over Time?

Set up a monthly cadence. Once a quarter is too infrequent for larger sites where URL count changes regularly.

Three metrics to track: average Googlebot requests per day (from GSC Crawl Stats), percentage of 200 vs 3xx vs 4xx responses (a rising 4xx percentage often indicates link rot or deleted pages still being crawled), and average server response time per Googlebot request. If any of those metrics degrades month over month, dig into what changed in your URL architecture.

Use the sitemap generator tool to maintain a clean, up-to-date sitemap that accurately reflects your indexable URL set. A sitemap that is out of sync with your actual content is one of the fastest ways to bleed crawl budget.

Frequently Asked Questions

Does Google guarantee it will crawl everything in my sitemap?

No. The sitemap is a suggestion, not a command. Google uses it as a discovery mechanism and a crawl priority signal, but it still applies its own crawl demand and rate limit logic. Submitting a URL to a sitemap does not guarantee it will be crawled or indexed. That said, a well-maintained sitemap consistently beats no sitemap for crawl coverage.

Can noindex hurt crawl budget?

Counterintuitively, yes. Googlebot still has to crawl a page to read the noindex directive. So if you have thousands of noindex pages in your architecture, they are still consuming crawl budget on discovery and re-visits. For pages you truly want excluded, blocking them in robots.txt at the discovery stage is more crawl-efficient, assuming you do not also need the noindex signal for pages that are already linked externally.

How long does it take to see improvement after fixing crawl budget issues?

In most cases, you will see changes in GSC Crawl Stats within 2-4 weeks of significant changes like blocking parameter URLs or removing a large batch of low-value pages. Index coverage improvements for your priority content often follow within 30-60 days, depending on how aggressively Googlebot re-crawls your site.

Crawl Budget: What It Is and How to Optimise It