Search Engine Crawl Log Analyzer
Summarise server log lines to show which pages search engines fetch, how often, what status codes they receive and how much crawl activity goes to pages you do not want indexed.
Last reviewed by the Radiatus Cloud team
Need this done properly for your business?
Radiatus delivers secure cloud, DevOps & compliance engineering.
Logs record what happened rather than what should have
Every other source of crawl information is a summary or a sample. Server logs are the actual record: which URL was requested, when, by which client, and what status was returned. That makes them the only way to answer several questions definitively, including whether a page has ever been fetched at all, how often the pages that matter are revisited, and how much activity is going to URLs you would rather were ignored.
The pattern of status codes is the finding
A crawler receiving a steady proportion of redirects is spending capacity on hops rather than content, which usually means internal links point at old URLs that redirect rather than at their destinations. Repeated server errors cause crawl rate to be reduced deliberately, so an error rate that looks tolerable to visitors can quietly suppress crawling. Not-found responses in volume normally mean links somewhere still point at pages that no longer exist.
Compare what is fetched against what matters
The most useful comparison is between the pages a crawler spends its time on and the pages that earn money. Sites frequently discover that most crawl activity goes to filtered listings, paginated archives and parameter variants while important pages are visited rarely. That comparison is not available from any tool that samples, and it is usually the finding that justifies looking at logs in the first place.
Related tools
- Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
- Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
- Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
- Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.
Frequently Asked Questions
What can logs tell me that other tools cannot?
Whether a page has ever been fetched, exactly how often each page is revisited, and what status code was returned each time. Everything else is a sample or a summary, so logs are the only definitive record.
Why does the status code mix matter?
Because redirects consume capacity without delivering content, and repeated server errors cause crawl rate to be reduced deliberately. An error rate that looks tolerable to visitors can quietly suppress how much of the site gets crawled.
How do I verify a crawler is genuine?
By reverse DNS on the requesting address and a forward lookup back to confirm it resolves to the search engine's own domain. The user agent string alone can be set to anything, so it proves nothing on its own. This tool groups by the declared agent and cannot verify it.
How much log data do I need?
At least a week to see a full crawl pattern, and a month for a large site where deep pages are revisited rarely. A single day mostly shows the pages crawled most often, which you probably already know.
What should I do about crawl activity on unimportant pages?
Work out why those URLs are being discovered, which is usually internal links or a sitemap listing them. Fixing the source is more effective than adding rules, since a blocked URL that is still linked continues to be discovered.
Privacy & Security
Everything runs in your browser; nothing is uploaded.
How to Use
Paste server log lines to see which pages search engines fetch and what they receive.
Disclaimer: This tool is provided "as is" without warranty of any kind. Results are for educational and utility purposes.
Related Tools
Meta Tag Analyzer
SEOCheck a page's meta tags for the problems that actually affect indexing and click-through.
Redirect Chain Checker
SEOTrace the full path of redirects (301/302) to find loops or lost link juice.
Robots.txt Validator
SEOValidate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.