# How Many Sites Use llms.txt? New Data

> We checked 355 top domains: 18.0% publish a real llms.txt. Counting HTTP 200s says 39.7% — a 2.2x overcount from soft 404s. Plus the crawler cross-tab.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How Many Sites Use llms.txt? New Data

# How Many Sites Use llms.txt? New Data

SEO dataSep 2, 2026·8 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [What we measured](/blog/how-many-sites-use-llms-txt/#what-we-measured) [The number, and why everyone else's is wrong](/blog/how-many-sites-use-llms-txt/#the-number-and-why-everyone-else-s-is-wrong) [What the real files look like](/blog/how-many-sites-use-llms-txt/#what-the-real-files-look-like) [Size does not predict adoption — which is unusual](/blog/how-many-sites-use-llms-txt/#size-does-not-predict-adoption-which-is-unusual) [The people who publish llms.txt do not block AI crawlers](/blog/how-many-sites-use-llms-txt/#the-people-who-publish-llms-txt-do-not-block-ai-crawlers) [The awkward part: nobody documents reading it](/blog/how-many-sites-use-llms-txt/#the-awkward-part-nobody-documents-reading-it) [What to do instead, in priority order](/blog/how-many-sites-use-llms-txt/#what-to-do-instead-in-priority-order) [Methodology and limits](/blog/how-many-sites-use-llms-txt/#methodology-and-limits)

64 of 355 top domains publish a real llms.txt — 18.0%. Count by HTTP status instead and you get 39.7%, because 77 of those sites answer 200 with their homepage or an error page. That 2.2x gap is why no two published adoption figures agree.

## What we measured

On 2 September 2026 we requested `/llms.txt` from the top 800 apex domains of the [Tranco research ranking](https://tranco-list.eu/) — the same corpus as our [AI crawler blocking study](https://quanticdata.io/blog/should-i-block-ai-crawlers/) and our [no-JavaScript audit](https://quanticdata.io/blog/do-ai-crawlers-render-javascript/), so all three cross-reference. 355 domains answered. For 309 of them we also had a parseable robots.txt, which lets us ask a question nobody has been able to ask before: do the sites that publish llms.txt actually let AI crawlers in?

Every per-domain result is published as [a CSV](https://quanticdata.io/blog/how-many-sites-use-llms-txt/llms-txt-adoption-study.csv?v=20260902).

## The number, and why everyone else's is wrong

The [llms.txt specification](https://llmstxt.org/) is precise about what the file is: an H1 with the site or project name — the only required section — optionally followed by a blockquote summary, then H2-delimited lists of markdown links to further detail. So "does this site have an llms.txt" is a content question, not a status-code question.

Almost every published adoption figure treats it as a status-code question, and that is where they break:

| Verdict | Domains | Share of 355 |
| --- | --- | --- |
| Real llms.txt | **64** | **18.0%** |
| — of which conform to the spec (H1 first) | 62 | 17.5% |
| HTTP 200, but the body is not an llms.txt | 77 | 21.7% |
| Honest 404 or 410 | 191 | 53.8% |
| Other status, unreachable, empty | 23 | 6.5% |
| *Naive count — anything answering 200* | *141* | *39.7%* |

Those 77 sites are the trap. A single-page app with a catch-all route returns its shell for any path; a CDN returns a branded error page with a 200; a localised site returns a redirect notice. We saw all three. Google's `/llms.txt` returns 172 bytes of Russian redirect copy. Reddit's returns the string "Page not found". Apple's returns the marketing text of its homepage. Every one of them counts as adoption if you only look at the status line.

The discriminator is trivial once you look: a real llms.txt is a list of markdown links. Of the 64 real files the median holds 74 links. Of the 77 impostors, every single one contains zero.

## What the real files look like

Adoption is one question; whether the files are any good is another.

- **62 of 64 open with an H1**, the spec's one hard requirement. The two that do not are large link dumps with no title at all.

- **53 of 64 include the blockquote summary.** This is the part an agent reads first to decide whether the rest is relevant, and one in six files omits it.

- **61 of 64 use H2 sections** to group their link lists.

- **Median size 13.7 KB, median 74 links.** That is a curated index, which is what the spec asks for.

- **Nine files exceed 100 KB**, and the largest is 1 MB with 3,772 links. At that size the file has stopped being an overview and become a sitemap with prose attached — the opposite of the intent, and more than most agents will read in one pass.

- **Only 11 of 64 also publish `llms-full.txt`**, the concatenated full-text companion.

The adopter list is what you would guess from the shape of the format: developer platforms and SaaS documentation. Stripe, GitHub, Shopify, Slack, Atlassian, Datadog, Zendesk, Okta, Mailchimp, Calendly, Intercom, Cursor. It is a documentation convention that has not yet left documentation.

## Size does not predict adoption — which is unusual

| Segment | Domains | Publish llms.txt |
| --- | --- | --- |
| Tranco top 100 | 25 | 20.0% |
| Rank 101 and below | 278 | 19.4% |

Flat. That is worth pausing on, because it is the opposite of what we found on the same corpus for AI-crawler blocking, where the top 100 blocked GPTBot at roughly twice the rate of everyone else. Blocking scales with how much you have to lose. Publishing an llms.txt does not scale with anything except whether your product has documentation.

## The people who publish llms.txt do not block AI crawlers

This is the cross-tab the two datasets make possible, and it is the clearest result in either study. Among the 309 domains where we have both files:

| Blocks at the site root | Publishes llms.txt (64) | Does not (245) |
| --- | --- | --- |
| GPTBot | **4.7%** | 20.4% |
| ClaudeBot | **3.1%** | 21.6% |
| Google-Extended | **4.7%** | 18.8% |
| CCBot | **7.8%** | 23.7% |
| PerplexityBot | **1.6%** | 14.3% |
| OAI-SearchBot | **0%** | 9.4% |
| Claude-SearchBot | **0%** | 10.6% |

Publishers block the training crawlers four to seven times less often than non-publishers, and **not one of the 64 blocks either retrieval crawler**. Only three sites contradict themselves by publishing a welcome mat and blocking a major crawler.

So llms.txt is not really a technical signal. It is a declaration of posture. The web is separating into a group that wants assistants to read it and says so twice — once in robots.txt by not blocking, once in llms.txt by curating — and a much larger group doing the opposite. Which camp a site is in is now predictable from a single file.

## The awkward part: nobody documents reading it

An honest study has to state this plainly. We checked the crawler documentation of every major operator — [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) and [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers). Every one specifies robots.txt tokens in detail. **None of them documents consuming llms.txt.** Several of those same companies publish one for their own documentation, which makes them authors of the convention rather than readers of it.

That does not make the file useless, and the spec is careful about the claim: llms.txt is meant to be used at inference time, on demand, when an agent needs orientation on a topic — not as a crawl directive. Agents that fetch URLs directly can and do request it, and a coding agent pointed at your docs will read a good index before it reads forty pages. But anyone telling you that publishing llms.txt gets you into ChatGPT's answers is describing a mechanism that no vendor has documented.

The defensible reasons to publish one: it costs an afternoon, it is the cheapest possible statement of posture, and it helps the agents that *do* fetch on demand. The indefensible reason is expecting it to substitute for being crawlable — which, on the evidence of the same corpus, is where the actual losses are.

## What to do instead, in priority order

1. **Be readable without JavaScript.** None of the major AI crawlers execute it. On this same corpus, 6.4% of top sites have readable content only with JS — a bigger hole than any file at the site root can patch.

2. **Check what your robots.txt actually resolves to per agent.** Blocking the retrieval crawlers is the most common self-inflicted wound we measured, and it is invisible until you test it.

3. **Then publish an llms.txt** — with the H1, the blockquote summary, and a curated list rather than a dump. If it is over 100 KB, it is a sitemap and something has gone wrong.

4. **Serve clean markdown at the URLs it points to.** An index pointing at pages an agent cannot parse moves the problem rather than solving it.

The [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/) covers the second point, and the [SEO audit API](https://quanticdata.io/seo-audit/) covers the first by fetching any URL twice — once as a plain HTTP client, once rendered — and returning the diff.

## Methodology and limits

One request per apex domain on 2 September 2026, plain HTTP, redirects followed. A file counted as real if it returned 200 and either opened with an H1 or carried at least ten markdown links; classification code and per-domain results are in the CSV so you can apply a stricter rule and recount.

Four limits worth stating. The corpus is the Tranco top 800, which skews heavily toward technology and SaaS — adoption across the whole web is certainly lower than 18%, and this figure should never be quoted as "of all websites". The spec allows llms.txt at any subpath, so a site with `/docs/llms.txt` and nothing at the root counts as a non-publisher here; root-only measurement undercounts. Some sites advertise the file through a `Link: rel="llms-txt"` header rather than at the conventional path, which we did not follow. And this is a single-day snapshot of a convention that is moving quickly.

### Sources & further reading

- [The /llms.txt file, v2 — the specification (Jeremy Howard, Answer.AI)](https://llmstxt.org/)

- [OpenAI — Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

- [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

- [Perplexity — Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

- [Google — List of Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

- [Tranco — a research-oriented top sites ranking](https://tranco-list.eu/)

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

## FAQ

Quick answers on how many sites use llms.txt.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### How many websites have an llms.txt file?

In our 2 September 2026 audit, 64 of 355 reachable top-Tranco domains published a real llms.txt — 18.0%, with 62 conforming to the spec by opening with an H1. Counting by HTTP status alone gives 141 domains, or 39.7%, because 77 sites return 200 with their homepage or an error page instead of a file. The corpus skews toward technology and SaaS, so adoption across the whole web is lower.

### Why do llms.txt adoption figures differ so much between studies?

Because most count HTTP 200 responses rather than checking the body. In our corpus that inflates the figure by 2.2x: 77 domains answer 200 at /llms.txt with a single-page-app shell, a branded error page or a redirect notice. The reliable discriminator is markdown links — the 64 real files hold a median of 74, and all 77 impostors contain zero.

### Do ChatGPT, Claude or Perplexity actually read llms.txt?

None of them documents doing so. OpenAI, Anthropic, Perplexity and Google all specify robots.txt user-agent tokens in detail, and none mentions consuming llms.txt — though several publish one for their own documentation. The spec positions the file as an inference-time aid for agents fetching on demand, not as a crawl directive, and that is the claim the evidence supports.

### Does publishing llms.txt help me get cited by AI assistants?

There is no documented mechanism by which it does. What our data shows is a correlation of posture, not of causation: sites publishing llms.txt block GPTBot at 4.7% against 20.4% for non-publishers, and none of them blocks OAI-SearchBot or Claude-SearchBot. Being crawlable is what makes citation possible; the file is a statement of intent alongside it.

### What does a valid llms.txt look like?

Per the spec, an H1 with the site or project name — the only required section — then optionally a blockquote summary, then optional markdown sections without headings, then H2-delimited lists of markdown links to further detail. In our sample 62 of 64 files open with the H1, 53 include the blockquote, and 61 use H2 sections.

### How big should an llms.txt be?

The median real file in our sample is 13.7 KB with 74 links, which reads as a curated index. Nine files exceed 100 KB and the largest is 1 MB with 3,772 links — at that point it is a sitemap with prose attached, which is the opposite of what the format is for. If yours is over 100 KB, split it: the spec allows a file at any subpath, and the most specific one wins.

### What is llms-full.txt?

A companion file holding the concatenated full text of the documents an llms.txt indexes, so an agent can take everything in one fetch instead of following dozens of links. It is far rarer than llms.txt itself: only 11 of the 64 publishers in our sample serve one.

## Measure your own site the way we measured theirs

The robots.txt tester resolves your file per named AI agent, and the SEO audit API fetches any URL twice — plain HTTP with no JavaScript and fully rendered — returning both views and the diff at $0.0012 per URL. Failed requests are never billed, and every account gets $2 of free API usage a month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore SEO Audit API](https://quanticdata.io/seo-audit/)

## Related reading

[SEO data Should I Block AI Crawlers? New Data We parsed the robots.txt of 326 of the web’s top domains. AI crawlers are blocked seven times more often than Googlebot, training bots twice as often as the search bots from the same company — and a large share of the blocking lands on crawlers that would have cited the site and never trained on it. Read →](https://quanticdata.io/blog/should-i-block-ai-crawlers/) [SEO data AI Crawler User Agent List A reference table of every AI user agent: which operator runs it, whether its job is training, retrieval or a user-triggered fetch, what blocking it actually costs you, and the share of 326 top domains that block it today. Plus the robots.txt mechanics that make these rules mean something other than intended. Read →](https://quanticdata.io/blog/ai-crawler-user-agent-list/) [SEO data llms.txt vs robots.txt robots.txt answers whether an agent may fetch a page and is honoured by every major AI operator. llms.txt answers what is worth reading and is documented as read by none of them. The formats, the resolution rules, what neither file can do, and measured adoption for both across the same 355 domains. Read →](https://quanticdata.io/blog/llms-txt-vs-robots-txt/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/how-many-sites-use-llms-txt/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt