Crawl Budget Optimization [Video]

Crawl budget is the number of requests Googlebot makes to your site in a given period. Put more bluntly, it is the number of pages Googlebot crawls and the time it spends on the site. Update (October 2026): Google defines crawl budget as the set of URLs that Google can and wants to crawl.

By optimising crawl budget we can help Googlebot discover our important content more easily, crawl more of it, and control the time it spends.

Google re-crawls sites at regular intervals.

Gary Illyes of Google has explained that Google builds a list of the site's URLs before the crawl begins. The best outcome is Googlebot crawling every page on that list.

Googlebot works through the list from top to bottom. URLs that receive many internal links, along with sitemaps, help Google build that list in the first place. While crawling the list, Googlebot can meet negative conditions such as 40x and 50x errors or too many 30x redirects. In those cases Googlebot lowers the crawl budget, crawls fewer pages and leaves the site.

You can find how many pages Googlebot crawls on your site each day, and the average time it spends per page, in Search Console. Update (October 2026): Search Console shows this data in the Crawl Stats report under Settings, as total crawl requests, total download size and average response time.

Why does optimising crawl budget matter?

Optimising crawl budget matters because Googlebot crawls only a certain number of pages on our site every day. Because it does not allocate us an unlimited budget, there are times when it cannot crawl everything. If the budget is limited, we need to make sure Googlebot crawls the pages that matter most. Update (October 2026): Google wrote its crawl budget guide for sites with over 1 million pages that change weekly or over 10,000 pages that change daily.

Take an "About us" page: if we link to it from every page and list it near the top of the sitemap, Googlebot may crawl it daily. But is "About us" a page we target organic traffic with? Do we want Googlebot crawling it and spending even a small slice of the budget there? No.

When optimising crawl budget, the aim is to get Googlebot to prioritise the quality content pages we actually want traffic on.

How can we optimise crawl budget?

We can optimise crawl budget with the three pieces of work below.

1. Site speed
2. Clearing out unnecessary pages and keeping them out of the index
3. Server analysis

Site speed optimisation

Site speed optimisation cuts the time Googlebot spends downloading a page and raises the number of pages it crawls. I will not go into how to improve site speed here. There are plenty of tools and articles on PageSpeed already.

Instead let me share an experience. A client of mine had content pages that needed updating every single day, and not a small number of them: more than 500 a day. Imagine changing the title of 500+ pages daily. When yesterday's index showed in Google's results, the click-through rate (CTR) was low. When the index was refreshed daily, the CTR doubled. That mattered enormously to us.

So we asked how we could raise PageSpeed and get Googlebot to crawl more pages. We went through every externally called resource and every file on the server that loads with the page, and optimised them. That work reduced Googlebot's "time spent downloading a page".

In the image below you can see the "time spent downloading a page" line falling. Look at the "pages crawled per day" line and you see it rising in inverse proportion.

To sum the chart up: when the time Googlebot spends per page falls, the number of pages it crawls rises.

Get rid of the unnecessary pages on your site

Unnecessary pages abuse crawl budget, on e-commerce sites in particular: pointless content pages, needless parameters, products that read as duplicates. Using noindex and canonical tag best practices is usually the healthiest way to keep those to a minimum.

Once you have removed the unnecessary pages and want them out of Google's index, gather them all into a file and build a temporary sitemap. Submit it to Google and you will speed up their removal from the index. Alongside that you can block the unnecessary pages in robots.txt. As a final step you can delete the temporary sitemap from Search Console. Update (October 2026): Google never sees a noindex tag on a page blocked by robots.txt, so a page that carries noindex must not be blocked by robots.txt.

Check your redirects

While clearing out unnecessary pages, redirects get added to the removed pages. If your site has hundreds of thousands of products or sub-pages, keeping track of redirected URLs becomes hard. You end up with situations like this: page A redirects to page B. A month later B redirects to C. As the number of redirects grows, so do the chains.

Try to keep those chains to a minimum. When B is redirected to C, for example, redirect A straight to C instead of letting it pass through B.

Do not use canonical tags on 404 pages

We mostly see canonical tags on 404 pages at companies developing their own software in-house. It does not come up with WordPress or the large e-commerce platforms, but using a canonical tag on 404 pages can get those 404s indexed. If you use a canonical on your 404 pages and no noindex, you can end up with pointless indexed pages like the ones below.

Analyse your crawl budget

You have optimised your crawl budget, so how do you measure it? Crawl budget analysis runs on server logs and looks for the answers to three questions.

• How often does Googlebot come to the page?
• Which page does it crawl most?
• How much data does it download while crawling page X?

To answer those three questions you need to analyse your server's daily logs. You can build tables in Excel and interpret them, but my recommendation is to use a tool.

Splunk, Logentries or Stackify will all do the job. Update (October 2026): the Logentries address (logentries.com) now redirects to the Rapid7 site, while the Splunk and Stackify sites are live.

Log analysis is a subject in its own right. There is a whole article in it. If there is demand, I will write one. Update (October 2026): I wrote it, see log file analysis for SEO.