Robots.txt Tester

The matching rule is not first match wins

The rule that applies is the longest matching path, not the first one in the file, and when an allow and a disallow of the same length both match, allow wins. This surprises people repeatedly: a file with Disallow slash followed by Allow slash blog does permit the blog, because the allow rule is longer for those URLs. Reading a robots file top to bottom and stopping at the first match gives the wrong answer in exactly the cases where the file is doing something interesting.

Only one group applies

A crawler uses the single most specific group matching its user agent and ignores every other group entirely, including the wildcard group. If Googlebot has its own section, the rules under the wildcard do not apply to it at all, even for paths its own section never mentions. Adding a specific group for one crawler therefore silently removes all the wildcard rules from that crawler, which is one of the most common ways a robots file stops doing what its author intended.

Disallow does not mean deindex

A blocked URL can still appear in search results if other pages link to it, because the crawler knows the URL exists without being permitted to fetch it. The listing shows without a description, since the content was never read. To keep a page out of the index it must be crawlable and carry a noindex directive, which means blocking it in robots actually prevents removal by making the noindex tag unreadable.

Related tools

• Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
• Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
• Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
• Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.