Crawl Budget Calculator
Estimate how long a full crawl of your site takes at a given crawl rate, how much capacity duplicate and parameter URLs consume, and whether crawl budget is actually your problem.
Last reviewed by the Radiatus Cloud team
Need this done properly for your business?
Radiatus delivers secure cloud, DevOps & compliance engineering.
Crawl budget matters far less often than it is discussed
For sites below roughly ten thousand pages, crawl budget is almost never the limiting factor. Pages are not indexed because they are not worth indexing, not because a crawler ran out of capacity. The concern becomes real on large sites, sites generating URLs faster than they can be crawled, and sites where faceted navigation has multiplied a few thousand products into millions of near duplicate addresses. Diagnosing it wrongly leads to elaborate work that changes nothing.
What actually determines the rate
Two things: how fast your server responds, and how much a search engine wants to crawl you. The first is under your control and directly observable, since a slow server causes crawl rate to be reduced deliberately to avoid overloading it. The second follows from how much the site matters, which cannot be adjusted directly. Improving server response time is therefore the one reliable lever, and it is also the one most often overlooked in favour of adjusting directives.
Waste is the fixable part
A crawler spending its capacity on parameter variants, session identifiers, printer friendly duplicates, faceted combinations and paginated archives is not spending it on pages you want indexed. The arithmetic is stark: four filters with five options each generate over six hundred URLs from one page of content. Cutting that waste with canonicals, robots rules and parameter handling frees capacity without needing a search engine to grant more.
Related tools
- Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
- Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
- Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
- Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.
Frequently Asked Questions
When is crawl budget a real problem?
On sites above roughly ten thousand pages, sites generating new URLs faster than they can be crawled, and sites where faceted navigation has multiplied a modest catalogue into millions of near duplicate addresses. Below that, unindexed pages are usually not worth indexing rather than uncrawled.
How do I find my actual crawl rate?
From server logs, counting requests from verified search engine crawlers per day, or from the crawl stats report in Search Console. An estimate based on anything else is guesswork.
Does a faster server increase crawl rate?
Yes, and it is the most reliable lever available. Crawl rate is deliberately reduced when a server responds slowly, to avoid overloading it, so reducing response time directly raises the ceiling.
Do noindex pages still consume crawl budget?
Yes. The page must be fetched for the directive to be read, so it costs a crawl every time. Blocking in robots.txt saves the crawl and prevents the noindex being seen at all, which is why the two cannot be combined.
Should I block parameters in robots.txt?
Only when those URLs need no indexing at all and nothing links to them externally. A canonical is usually better, since it consolidates signals to the main URL, while a robots block leaves them isolated.
Privacy & Security
Everything runs in your browser; nothing is uploaded.
How to Use
Enter your page count and observed crawl rate to see how long a full pass takes.
Disclaimer: This tool is provided "as is" without warranty of any kind. Results are for educational and utility purposes.
Related Tools
Meta Tag Analyzer
SEOCheck a page's meta tags for the problems that actually affect indexing and click-through.
Redirect Chain Checker
SEOTrace the full path of redirects (301/302) to find loops or lost link juice.
Robots.txt Validator
SEOValidate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.