Robots.txt Analyzer

Understand what your robots.txt actually allows

A robots.txt file tells crawlers which parts of a site they may and may not fetch, and a single wrong line can hide a whole site from search or expose an area you meant to keep private. This analyser reads a robots.txt and explains its directives, so you can see what it really permits rather than what you think it says.

The rules are subtler than they look

Robots.txt syntax has traps. Directives apply per user-agent group, so a rule under one crawler’s name does not affect another. Allow and Disallow interact by specificity, not order, in modern crawlers. A stray Disallow of a single slash blocks the entire site. And robots.txt controls crawling, not indexing: a disallowed page can still appear in results if it is linked elsewhere. Reading the file correctly means understanding all of this.

A file worth getting right

Because robots.txt is one of the first things a crawler reads and governs access to everything, an error in it has site-wide consequences. Analysing it, especially after an edit, confirms you have blocked what you meant to and nothing more. The analysis runs in your browser, so your file is not uploaded.

Related tools

• Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
• Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
• Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
• Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.