Robots.txt Validator

Robots.txt controls crawling, not indexing

This is the distinction that causes the most damage. Disallow stops a compliant crawler fetching the page; it does not remove the page from search results. A blocked URL with external links pointing at it can still be indexed, appearing with no description because the crawler was never allowed to read it. To keep a page out of the index you need a noindex meta tag or header — which requires the page to be crawlable. Blocking a page you also want deindexed guarantees the opposite of the intent.

Specificity beats order

Google resolves conflicting rules by the length of the matching path, not by which appears first. Given Disallow: /admin/ and Allow: /admin/public/, the longer Allow wins for URLs under /admin/public/. When two rules are the same length, the least restrictive wins. Writing rules in a particular order to get a particular result is a misunderstanding.

User-agent groups do not merge

A crawler obeys exactly one group: the most specific one matching its name. If there is a Googlebot group and a wildcard group, Googlebot reads only the Googlebot group and ignores every rule under the wildcard, including ones you assumed applied to everyone. Rules meant for all crawlers must be repeated in each named group.

Wildcards and the end-of-string anchor

An asterisk matches any sequence and a dollar sign anchors to the end of the URL. Disallow: /*.pdf$ blocks PDFs but not /file.pdf?download=1, because the query string means the URL no longer ends in .pdf. Query parameters routinely defeat rules written without them in mind.

Never block CSS and JavaScript

Google renders pages to evaluate them. Blocking the stylesheets and scripts means it renders a broken page and judges it accordingly, including for mobile usability. The old habit of disallowing /assets/ or /wp-includes/ actively damages rankings on a modern site.

Location and scope are strict

The file must be at the root of the host, served as plain text, and it applies only to that exact host and protocol. A robots.txt at the www subdomain does not govern the apex, and the HTTPS file does not govern HTTP. A 404 means everything is allowed; a 500 means Google may treat the whole site as disallowed, which makes robots.txt availability a genuine uptime concern.