llms.txt vs robots.txt

They are not competing standards and one does not replace the other. robots.txt answers "may you fetch this?" and is honoured by every major AI crawler. llms.txt answers "what here is worth reading?" and, as of today, no AI operator documents reading it. Only one of the two can cost you traffic if you get it wrong.

One is permission, the other is orientation

The asymmetry in that last row of consequences is the practical point. A misconfigured robots.txt removes you from an assistant's answers outright. A missing llms.txt removes a convenience.

What robots.txt actually does, precisely

It is a set of groups, each headed by one or more User-agent lines and followed by Allow and Disallow rules. Three resolution rules decide everything, and all three routinely surprise people:

• Consecutive User-agent lines form one group sharing the rules beneath them. A blank line in the wrong place silently splits it.
• The most specific matching group wins outright. An agent with its own group ignores * entirely — so a site-wide Disallow: / under * does not apply to any bot you named elsewhere, even to allow it something trivial.
• Within a group the longest matching path wins, and Allow beats Disallow on an exact tie.

The AI-specific part is that the operators split their crawlers by job. GPTBot and ClaudeBot train; OAI-SearchBot and Claude-SearchBot build the index an assistant searches at answer time; PerplexityBot only ever indexes and, by Perplexity's own documentation, never trains. Blocking across that line is the most expensive mistake in the file — the full breakdown is in the AI crawler user agent list.

What llms.txt actually does, precisely

Per the specification, in this order: an optional byte-order mark, then an H1 with the site or project name — the only required section, then a blockquote summarising the project, then any markdown sections that are not headings, then zero or more H2-delimited "file lists" where each entry is a markdown link optionally followed by : and a note.

# Acme

> Acme is a payments API for marketplaces. Everything below is
> developer documentation; pricing and legal live on the main site.

## Docs

- [Quickstart](https://acme.com/docs/start.md): create a key and take a payment
- [API reference](https://acme.com/docs/api.md): every endpoint and parameter

## Optional

- [Changelog](https://acme.com/changelog.md)

Two details people miss. It may live at any subpath, and an agent should use the most specific file that applies — so a large site can ship /docs/llms.txt and /api/llms.txt rather than one enormous root file. And the companion llms-full.txt holds the concatenated full text so an agent can take everything in one fetch; it is rare, appearing on only 11 of the 64 publishers we measured.

Neither file does what people hope it does

robots.txt is not enforcement. It is a published preference that well-behaved crawlers honour. The user-triggered fetchers are explicitly outside it: OpenAI states that because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply, and Perplexity states that Perplexity-User generally ignores robots.txt. If a block has to hold, it belongs at the edge, not in a text file.

llms.txt is not access control. It grants nothing and forbids nothing. Listing a URL there does not make it fetchable if robots.txt disallows it, and omitting a URL does not hide it. A site that blocks GPTBot and publishes a beautiful llms.txt has blocked GPTBot.

And llms.txt is not a documented route into AI answers. We checked the crawler documentation of OpenAI, Anthropic, Perplexity and Google: all four specify robots.txt tokens in detail and none mentions consuming llms.txt — while several publish one for their own docs, making them authors of the convention rather than readers of it. The spec itself is careful here, positioning the file as an inference-time aid for agents fetching on demand rather than a crawl directive.

What the two files look like in the wild

We measured both across the same 355 top-Tranco domains on 2 September 2026.

Joining the two datasets gives the result that ties them together. Sites publishing llms.txt block GPTBot at 4.7% against 20.4% for everyone else, ClaudeBot at 3.1% against 21.6% — and none of the 64 blocks OAI-SearchBot or Claude-SearchBot. The two files are not alternatives; in practice they are two expressions of the same decision. The full numbers and per-domain data are in the llms.txt adoption study and the crawler blocking study.

Which to write first

1. Fix robots.txt. Resolve it per agent rather than reading it by eye, and check specifically that OAI-SearchBot, Claude-SearchBot and PerplexityBot are not caught by a rule aimed at training. This is the only step with a measurable downside if skipped.
2. Make sure the pages render without JavaScript. No major AI crawler executes it, and permission is worthless if the response body is an empty shell.
3. Then write llms.txt. H1, blockquote, curated H2 lists. Keep it an index — the median real file is 13.7 KB; anything past 100 KB has become a sitemap.
4. Point it at parseable URLs. Markdown twins of your pages beat linking to JavaScript-heavy HTML.

Step one is the robots.txt tester, step two the SEO audit API, which fetches a URL as a plain HTTP client and as a rendered browser and returns the difference.