How Many Sites Use llms.txt? New Data

64 of 355 top domains publish a real llms.txt — 18.0%. Count by HTTP status instead and you get 39.7%, because 77 of those sites answer 200 with their homepage or an error page. That 2.2x gap is why no two published adoption figures agree.

What we measured

On 2 September 2026 we requested /llms.txt from the top 800 apex domains of the Tranco research ranking — the same corpus as our AI crawler blocking study and our no-JavaScript audit, so all three cross-reference. 355 domains answered. For 309 of them we also had a parseable robots.txt, which lets us ask a question nobody has been able to ask before: do the sites that publish llms.txt actually let AI crawlers in?

Every per-domain result is published as a CSV.

The number, and why everyone else's is wrong

The llms.txt specification is precise about what the file is: an H1 with the site or project name — the only required section — optionally followed by a blockquote summary, then H2-delimited lists of markdown links to further detail. So "does this site have an llms.txt" is a content question, not a status-code question.

Almost every published adoption figure treats it as a status-code question, and that is where they break:

Verdict	Domains	Share of 355
Real llms.txt	64	18.0%
— of which conform to the spec (H1 first)	62	17.5%
HTTP 200, but the body is not an llms.txt	77	21.7%
Honest 404 or 410	191	53.8%
Other status, unreachable, empty	23	6.5%
Naive count — anything answering 200	141	39.7%

Those 77 sites are the trap. A single-page app with a catch-all route returns its shell for any path; a CDN returns a branded error page with a 200; a localised site returns a redirect notice. We saw all three. Google's /llms.txt returns 172 bytes of Russian redirect copy. Reddit's returns the string "Page not found". Apple's returns the marketing text of its homepage. Every one of them counts as adoption if you only look at the status line.

The discriminator is trivial once you look: a real llms.txt is a list of markdown links. Of the 64 real files the median holds 74 links. Of the 77 impostors, every single one contains zero.

What the real files look like

Adoption is one question; whether the files are any good is another.

• 62 of 64 open with an H1, the spec's one hard requirement. The two that do not are large link dumps with no title at all.
• 53 of 64 include the blockquote summary. This is the part an agent reads first to decide whether the rest is relevant, and one in six files omits it.
• 61 of 64 use H2 sections to group their link lists.
• Median size 13.7 KB, median 74 links. That is a curated index, which is what the spec asks for.
• Nine files exceed 100 KB, and the largest is 1 MB with 3,772 links. At that size the file has stopped being an overview and become a sitemap with prose attached — the opposite of the intent, and more than most agents will read in one pass.
• Only 11 of 64 also publish llms-full.txt, the concatenated full-text companion.

The adopter list is what you would guess from the shape of the format: developer platforms and SaaS documentation. Stripe, GitHub, Shopify, Slack, Atlassian, Datadog, Zendesk, Okta, Mailchimp, Calendly, Intercom, Cursor. It is a documentation convention that has not yet left documentation.

Size does not predict adoption — which is unusual

Segment	Domains	Publish llms.txt
Tranco top 100	25	20.0%
Rank 101 and below	278	19.4%

Flat. That is worth pausing on, because it is the opposite of what we found on the same corpus for AI-crawler blocking, where the top 100 blocked GPTBot at roughly twice the rate of everyone else. Blocking scales with how much you have to lose. Publishing an llms.txt does not scale with anything except whether your product has documentation.

The people who publish llms.txt do not block AI crawlers

This is the cross-tab the two datasets make possible, and it is the clearest result in either study. Among the 309 domains where we have both files:

Blocks at the site root	Publishes llms.txt (64)	Does not (245)
GPTBot	4.7%	20.4%
ClaudeBot	3.1%	21.6%
Google-Extended	4.7%	18.8%
CCBot	7.8%	23.7%
PerplexityBot	1.6%	14.3%
OAI-SearchBot	0%	9.4%
Claude-SearchBot	0%	10.6%

Publishers block the training crawlers four to seven times less often than non-publishers, and not one of the 64 blocks either retrieval crawler. Only three sites contradict themselves by publishing a welcome mat and blocking a major crawler.

So llms.txt is not really a technical signal. It is a declaration of posture. The web is separating into a group that wants assistants to read it and says so twice — once in robots.txt by not blocking, once in llms.txt by curating — and a much larger group doing the opposite. Which camp a site is in is now predictable from a single file.

The awkward part: nobody documents reading it

An honest study has to state this plainly. We checked the crawler documentation of every major operator — OpenAI, Anthropic, Perplexity and Google. Every one specifies robots.txt tokens in detail. None of them documents consuming llms.txt. Several of those same companies publish one for their own documentation, which makes them authors of the convention rather than readers of it.

That does not make the file useless, and the spec is careful about the claim: llms.txt is meant to be used at inference time, on demand, when an agent needs orientation on a topic — not as a crawl directive. Agents that fetch URLs directly can and do request it, and a coding agent pointed at your docs will read a good index before it reads forty pages. But anyone telling you that publishing llms.txt gets you into ChatGPT's answers is describing a mechanism that no vendor has documented.

The defensible reasons to publish one: it costs an afternoon, it is the cheapest possible statement of posture, and it helps the agents that do fetch on demand. The indefensible reason is expecting it to substitute for being crawlable — which, on the evidence of the same corpus, is where the actual losses are.

What to do instead, in priority order

1. Be readable without JavaScript. None of the major AI crawlers execute it. On this same corpus, 6.4% of top sites have readable content only with JS — a bigger hole than any file at the site root can patch.
2. Check what your robots.txt actually resolves to per agent. Blocking the retrieval crawlers is the most common self-inflicted wound we measured, and it is invisible until you test it.
3. Then publish an llms.txt — with the H1, the blockquote summary, and a curated list rather than a dump. If it is over 100 KB, it is a sitemap and something has gone wrong.
4. Serve clean markdown at the URLs it points to. An index pointing at pages an agent cannot parse moves the problem rather than solving it.

The robots.txt tester covers the second point, and the SEO audit API covers the first by fetching any URL twice — once as a plain HTTP client, once rendered — and returning the diff.

Methodology and limits

One request per apex domain on 2 September 2026, plain HTTP, redirects followed. A file counted as real if it returned 200 and either opened with an H1 or carried at least ten markdown links; classification code and per-domain results are in the CSV so you can apply a stricter rule and recount.

Four limits worth stating. The corpus is the Tranco top 800, which skews heavily toward technology and SaaS — adoption across the whole web is certainly lower than 18%, and this figure should never be quoted as "of all websites". The spec allows llms.txt at any subpath, so a site with /docs/llms.txt and nothing at the root counts as a non-publisher here; root-only measurement undercounts. Some sites advertise the file through a Link: rel="llms-txt" header rather than at the conventional path, which we did not follow. And this is a single-day snapshot of a convention that is moving quickly.