# llms.txt vs robots.txt

> One controls access and every AI operator honours it; the other is a markdown index none of them documents reading. What each does, and which to fix first.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/llms.txt vs robots.txt

# llms.txt vs robots.txt

SEO dataSep 2, 2026·6 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [One is permission, the other is orientation](/blog/llms-txt-vs-robots-txt/#one-is-permission-the-other-is-orientation) [What robots.txt actually does, precisely](/blog/llms-txt-vs-robots-txt/#what-robots-txt-actually-does-precisely) [What llms.txt actually does, precisely](/blog/llms-txt-vs-robots-txt/#what-llms-txt-actually-does-precisely) [Neither file does what people hope it does](/blog/llms-txt-vs-robots-txt/#neither-file-does-what-people-hope-it-does) [What the two files look like in the wild](/blog/llms-txt-vs-robots-txt/#what-the-two-files-look-like-in-the-wild) [Which to write first](/blog/llms-txt-vs-robots-txt/#which-to-write-first)

They are not competing standards and one does not replace the other. robots.txt answers "may you fetch this?" and is honoured by every major AI crawler. llms.txt answers "what here is worth reading?" and, as of today, no AI operator documents reading it. Only one of the two can cost you traffic if you get it wrong.

## One is permission, the other is orientation

|  | robots.txt | llms.txt |
| --- | --- | --- |
| Question it answers | May this agent fetch this path? | What on this site is worth reading, and where is it? |
| Format | Plain-text directives in user-agent groups | Markdown: H1, blockquote summary, H2 lists of links |
| Standardised as | RFC 9309, an IETF standard since 2022 | A proposal at llmstxt.org, currently v2 |
| Read at | Crawl time, before the fetch | Inference time, on demand, when an agent needs orientation |
| Documented consumers | OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl | None of them |
| Scope | The whole host | Its own path — `/docs/llms.txt` covers `/docs`, most specific wins |
| Getting it wrong costs you | Every citation from the agents you blocked | Very little either way |
| Adoption in the top 800 | Near-universal — 326 of 362 serve one | 18.0% — 64 of 355 |

The asymmetry in that last row of consequences is the practical point. A misconfigured robots.txt removes you from an assistant's answers outright. A missing llms.txt removes a convenience.

## What robots.txt actually does, precisely

It is a set of groups, each headed by one or more `User-agent` lines and followed by `Allow` and `Disallow` rules. Three resolution rules decide everything, and all three routinely surprise people:

- **Consecutive `User-agent` lines form one group** sharing the rules beneath them. A blank line in the wrong place silently splits it.

- **The most specific matching group wins outright.** An agent with its own group ignores `*` entirely — so a site-wide `Disallow: /` under `*` does not apply to any bot you named elsewhere, even to allow it something trivial.

- **Within a group the longest matching path wins**, and `Allow` beats `Disallow` on an exact tie.

The AI-specific part is that the operators split their crawlers by job. GPTBot and ClaudeBot train; OAI-SearchBot and Claude-SearchBot build the index an assistant searches at answer time; PerplexityBot only ever indexes and, by Perplexity's own documentation, never trains. Blocking across that line is the most expensive mistake in the file — the full breakdown is in [the AI crawler user agent list](https://quanticdata.io/blog/ai-crawler-user-agent-list/).

## What llms.txt actually does, precisely

Per the [specification](https://llmstxt.org/), in this order: an optional byte-order mark, then an **H1 with the site or project name — the only required section**, then a blockquote summarising the project, then any markdown sections that are not headings, then zero or more H2-delimited "file lists" where each entry is a markdown link optionally followed by `:` and a note.

```
# Acme

> Acme is a payments API for marketplaces. Everything below is
> developer documentation; pricing and legal live on the main site.

## Docs

- [Quickstart](https://acme.com/docs/start.md): create a key and take a payment
- [API reference](https://acme.com/docs/api.md): every endpoint and parameter

## Optional

- [Changelog](https://acme.com/changelog.md)
```

Two details people miss. It may live at any subpath, and an agent should use the most specific file that applies — so a large site can ship `/docs/llms.txt` and `/api/llms.txt` rather than one enormous root file. And the companion `llms-full.txt` holds the concatenated full text so an agent can take everything in one fetch; it is rare, appearing on only 11 of the 64 publishers we measured.

## Neither file does what people hope it does

**robots.txt is not enforcement.** It is a published preference that well-behaved crawlers honour. The user-triggered fetchers are explicitly outside it: OpenAI states that because ChatGPT-User actions are initiated by a person, robots.txt rules may not apply, and Perplexity states that Perplexity-User generally ignores robots.txt. If a block has to hold, it belongs at the edge, not in a text file.

**llms.txt is not access control.** It grants nothing and forbids nothing. Listing a URL there does not make it fetchable if robots.txt disallows it, and omitting a URL does not hide it. A site that blocks GPTBot and publishes a beautiful llms.txt has blocked GPTBot.

**And llms.txt is not a documented route into AI answers.** We checked the crawler documentation of OpenAI, Anthropic, Perplexity and Google: all four specify robots.txt tokens in detail and none mentions consuming llms.txt — while several publish one for their own docs, making them authors of the convention rather than readers of it. The spec itself is careful here, positioning the file as an inference-time aid for agents fetching on demand rather than a crawl directive.

## What the two files look like in the wild

We measured both across the same 355 top-Tranco domains on 2 September 2026.

| Measured | Result |
| --- | --- |
| Serve a parseable robots.txt | 326 of 362 |
| Block Googlebot at the root | 2.5% |
| Block GPTBot at the root | 17.8% |
| Block GPTBot while allowing Googlebot | 15.3% |
| Publish a real llms.txt | 64 of 355 — 18.0% |
| Return HTTP 200 at /llms.txt without serving one | 77 — which inflates naive counts 2.2x |

Joining the two datasets gives the result that ties them together. Sites publishing llms.txt block GPTBot at 4.7% against 20.4% for everyone else, ClaudeBot at 3.1% against 21.6% — and **none of the 64 blocks OAI-SearchBot or Claude-SearchBot**. The two files are not alternatives; in practice they are two expressions of the same decision. The full numbers and per-domain data are in [the llms.txt adoption study](https://quanticdata.io/blog/how-many-sites-use-llms-txt/) and [the crawler blocking study](https://quanticdata.io/blog/should-i-block-ai-crawlers/).

## Which to write first

1. **Fix robots.txt.** Resolve it per agent rather than reading it by eye, and check specifically that OAI-SearchBot, Claude-SearchBot and PerplexityBot are not caught by a rule aimed at training. This is the only step with a measurable downside if skipped.

2. **Make sure the pages render without JavaScript.** No major AI crawler executes it, and permission is worthless if the response body is an empty shell.

3. **Then write llms.txt.** H1, blockquote, curated H2 lists. Keep it an index — the median real file is 13.7 KB; anything past 100 KB has become a sitemap.

4. **Point it at parseable URLs.** Markdown twins of your pages beat linking to JavaScript-heavy HTML.

Step one is the [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/), step two the [SEO audit API](https://quanticdata.io/seo-audit/), which fetches a URL as a plain HTTP client and as a rendered browser and returns the difference.

### Sources & further reading

- [The /llms.txt file, v2 — the specification](https://llmstxt.org/)

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

- [Google — How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

- [OpenAI — Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

- [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

- [Perplexity — Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

- [Google — List of Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

## FAQ

Quick answers on llms.txt vs robots.txt.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What is the difference between llms.txt and robots.txt?

robots.txt is an access-control convention standardised as RFC 9309: it tells crawlers which paths they may fetch, per user agent, and every major AI operator documents honouring it. llms.txt is a markdown index proposed at llmstxt.org that tells an agent what is worth reading and where, read on demand at inference time. One governs permission, the other orientation.

### Does llms.txt replace robots.txt?

No, and it cannot. llms.txt grants no access and forbids none — listing a URL there does not override a Disallow, and omitting one hides nothing. A site that blocks GPTBot in robots.txt and publishes a perfect llms.txt has still blocked GPTBot.

### Do I need both files?

You need robots.txt to be correct, because getting it wrong removes you from AI answers. llms.txt is optional and cheap: 18.0% of top domains publish one in our measurement, overwhelmingly developer platforms and SaaS documentation. Fix the first before writing the second.

### Is llms.txt an official standard?

No. It is a proposal published at llmstxt.org, currently at version 2, and it has no standards-body status. robots.txt by contrast has been an IETF standard since 2022 as RFC 9309. That difference shows in adoption: near-universal for robots.txt against 18.0% for llms.txt among the top domains we measured.

### Do AI crawlers read llms.txt?

None of the major operators documents doing so. OpenAI, Anthropic, Perplexity and Google all specify their robots.txt user-agent tokens in detail and none mentions consuming llms.txt, though several publish one for their own documentation. The spec positions it as an inference-time aid for agents that fetch on demand, which is a narrower claim than most articles make for it.

### Where should llms.txt be placed?

At /llms.txt for the whole site, or at any subpath to cover the URLs beneath it — /docs/llms.txt covers /docs. Where more than one file applies, an agent should use the most specific. That makes splitting a large index by section preferable to one enormous root file.

### What is llms-full.txt and do I need it?

It holds the concatenated full text of the documents your llms.txt indexes, so an agent can take everything in a single fetch rather than following dozens of links. It is optional and uncommon — only 11 of the 64 llms.txt publishers in our sample serve one.

## Resolve both files before you rewrite either

The free robots.txt tester shows which rule actually decides for each named AI agent, and the SEO audit API fetches any URL twice — plain HTTP with no JavaScript and fully rendered — returning both views and the diff at $0.0012 per URL. Failed requests are never billed, and every account gets $2 of free usage a month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore SEO Audit API](https://quanticdata.io/seo-audit/)

## Related reading

[SEO data Should I Block AI Crawlers? New Data We parsed the robots.txt of 326 of the web’s top domains. AI crawlers are blocked seven times more often than Googlebot, training bots twice as often as the search bots from the same company — and a large share of the blocking lands on crawlers that would have cited the site and never trained on it. Read →](https://quanticdata.io/blog/should-i-block-ai-crawlers/) [SEO data AI Crawler User Agent List A reference table of every AI user agent: which operator runs it, whether its job is training, retrieval or a user-triggered fetch, what blocking it actually costs you, and the share of 326 top domains that block it today. Plus the robots.txt mechanics that make these rules mean something other than intended. Read →](https://quanticdata.io/blog/ai-crawler-user-agent-list/) [SEO data How Many Sites Use llms.txt? New Data 64 of 355 top domains publish a real llms.txt. Count by status code instead and you get 39.7%, because 77 sites answer 200 with their homepage. Includes what the real files contain, and the cross-tab nobody has run: publishers of llms.txt block AI crawlers four to seven times less, and none of them blocks a retrieval crawler. Read →](https://quanticdata.io/blog/how-many-sites-use-llms-txt/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/llms-txt-vs-robots-txt/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt