Part of the The Complete Technical SEO Guide guide.
The number of URLs a search engine will fetch from a site over a period, set by server capacity and the engine's demand for the content.
Also known as crawl allocation, crawl capacity.
Crawl budget is the number of URLs a search engine is willing and able to fetch from a site over a given period. It is shaped by how fast the server responds and how much the engine wants the content. Wasting it on duplicates, parameters and redirects delays discovery of the pages that matter.
Key takeaways
- 01Crawl budget is set by crawl capacity, which your server controls, and crawl demand, which your content controls.
- 02Only large or frequently changing sites are truly budget-limited, but waste hurts sites of every size.
- 03Server logs, not crawler simulations, show how the budget is actually being spent.
Where the term comes from
Search engines fetch pages with a fleet of crawlers that has to be shared across the whole web. Google's documentation describes the share any one site receives as the product of two limits. Crawl capacity is the rate the crawler believes the server can sustain without slowing down for real users. Crawl demand is how much the engine wants to fetch, driven by how popular the URLs are and how often they change. The combination is what practitioners call crawl budget.
How it is measured
The only direct measurement is the server log. Every request from Googlebot or Bingbot is recorded with the URL, the status code and the time taken, so the log shows how many fetches the site received, which URLs consumed them, and how many were wasted on redirects, parameters or pages that return errors. Search Console's crawl stats report gives a summarised view for Google alone. Our article on what crawl budget is and why it matters walks through both sources and the fixes that follow.
When it matters
A brochure site with a hundred pages will rarely exhaust its budget. A retailer with faceted navigation, a publisher with archives, or any site with session parameters in URLs can generate millions of crawlable addresses, and then the budget decides which of them ever reach the index. The technical SEO guide treats crawling as the first stage of the pipeline for that reason.
Related
Blog
What Is Crawl Budget and Why It MattersCrawl budget explained for site owners. What it is, when it actually limits your site, and the fixes that stop search engines wasting crawls.
Comparisons
Semrush vs Ahrefs for Australian SMBsSemrush and Ahrefs compared on keyword data, site auditing, backlinks, rank tracking, local features and price for Australian small and mid-sized businesses.
Tools
SEO ROI CalculatorA free calculator that turns your organic traffic, expected uplift, conversion rate and lead value into extra leads, revenue and return on SEO spend.
FAQs
Technical SEO Frequently Asked QuestionsStraight answers to the technical SEO questions Australian businesses ask most, from crawl budget and canonicals to structured data and speed.