LLMS.txt Generator

What llms.txt proposes

A markdown file at /llms.txt listing a site's most useful pages with short descriptions, so a language model with a limited context window reads the right material rather than crawling navigation, cookie banners and boilerplate. It is a proposed convention rather than a ratified standard, and adoption by AI vendors is partial.

It is not robots.txt

robots.txt restricts crawling. llms.txt does the opposite — it recommends. It carries no directives, blocks nothing, and confers no permission. A page listed in llms.txt but disallowed in robots.txt is still disallowed, and the two files answer different questions.

Format

An H1 with the site name, an optional blockquote summary, then H2 sections containing markdown link lists with a short description after each colon. An optional section is conventionally used for material a model may skip when short of context. Plain markdown, no front matter.

Curate rather than dump

The value is selection. A file listing every URL on the site reproduces the sitemap and defeats the purpose, since the point is to spend a limited context window on the pages that actually answer questions. Twenty well-chosen entries beat a thousand.

Worth doing despite uncertain adoption

The cost is one small file. The genuine work is upstream: an AI system can only cite content that exists, is crawlable and answers a question directly. A site whose pages are thin gains nothing from listing them, which makes llms.txt the last step in an AI-visibility effort rather than the first.