# Crawl Budget

The number of pages Googlebot will crawl and index on your site within a given timeframe, determined by crawl rate and crawl demand.

Crawl budget refers to the number of URLs Googlebot (or other search engine crawlers) will fetch from your site in a set period. For most small-to-medium sites, crawl budget is rarely a problem — but for large sites with thousands of pages, mismanaging it can mean important pages go unindexed.

## Two Components

- **Crawl rate limit** — how fast Googlebot can crawl without overwhelming your server. Google adjusts this based on your server’s response times and the crawl limit you set in Search Console.
- **Crawl demand** — how much Google _wants_ to crawl your site, driven by popularity (backlinks, traffic) and staleness (how often content changes).

## When Crawl Budget Matters

Crawl budget becomes a concern when you have:

- Thousands of low-quality or duplicate URLs (faceted navigation, session IDs, print versions)
- Pages blocked incorrectly by robots.txt or noindex
- Very large sites where new or updated content is slow to get indexed

## How to Optimise Crawl Budget

- Block low-value URLs via robots.txt (admin pages, search result pages, duplicate parameters)
- Use canonical tags to consolidate duplicate content
- Keep your sitemap clean and up to date
- Reduce redirect chains — each hop costs crawl budget
- Improve server response times (faster servers = higher crawl rate limit)

🎉 Use code BLACKFRIDAY2025 to get 30% off — valid until Dec 1, 23:59!