Robots.txt Generator

What robots.txt does and does not do

robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, because the crawler never fetched the page to discover a noindex tag. That is the single most misunderstood aspect of the file. To keep a page out of the index, allow crawling and serve noindex; to keep it out entirely, require authentication.

Matching rules

Directives are prefix matches. Disallow: /api blocks /api, /api/v1 and also /api-documentation, which is rarely intended. Add the trailing slash when you mean a directory. Two wildcards are widely supported: * matches any sequence and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs specifically. Where an allow and a disallow both match, the more specific rule wins, and ties go to allow.

User-agent groups do not merge

A crawler obeys exactly one group: the most specific one matching its name. If you write a User-agent: * group with several disallows and then a User-agent: Googlebot group with one rule, Googlebot follows only that one rule and ignores the wildcard group entirely. Every directive you want a named crawler to obey has to be repeated inside its own group.

AI crawlers

Retrieval crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fetch pages so assistants can cite them, which is a distribution channel. Bulk corpus crawlers such as CCBot mainly feed training sets and send no traffic back. Deciding these separately is reasonable. Note that Google-Extended governs Gemini training only; it has no effect on AI Overviews or on ordinary Search, both of which follow Googlebot.

Practical rules

The file must sit at the domain root and is per host and per protocol, so subdomains need their own. Reference your sitemap with an absolute URL. Never list admin paths you actually want hidden, since robots.txt is public and reads as a map of where to look. And test before deploying: a stray Disallow: / deindexes an entire site, and it has happened to very large companies.

Related tools

• Meta Tag Analyzer — Check a page's meta tags for the problems that actually affect indexing and click-through.
• Redirect Chain Checker — Trace the full path of redirects (301/302) to find loops or lost link juice.
• Robots.txt Validator — Validate robots.txt syntax and check whether a specific URL is allowed or blocked for a given crawler.
• Sitemap Generator (Lite) — Generate a valid XML sitemap with lastmod dates and correct structure, and learn which URLs belong in it and which do not.