AI Crawler User Agent List

There are three kinds of AI user agent, and confusing them is the most expensive mistake in robots.txt today. Training crawlers build models. Retrieval crawlers decide whether an assistant can cite you. User-triggered fetchers act on a person's request and are documented as possibly ignoring robots.txt entirely. Here is every token, what it does, and what blocking it costs.

The complete list

Every entry below comes from the operator's own documentation. The last column is our own measurement: the share of 326 top Tranco domains that block that agent at the site root, audited on 2 September 2026 — the dataset behind should I block AI crawlers.

Two tokens in that table are not crawlers at all. Google-Extended has no HTTP user agent of its own — Google states the crawling is done with existing Google agents and the token is used purely as a control. Applebot-Extended likewise does not crawl; Apple describes it as only determining how data already collected by Applebot may be used. You cannot see either in your access logs, which is why people wrongly conclude they are not working.

The three jobs, and why the distinction is the whole point

Training crawlers collect content that may go into a model's training data. Blocking them is a copyright-and-licensing decision. It has little bearing on whether an assistant can quote you today: when an assistant cites a source with a link, it is almost always because a retrieval crawler indexed that page, not because the content was in a training set.

Retrieval crawlers build the index an assistant searches at answer time. These are the ones that decide whether you get cited. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot prevents indexing and may reduce visibility. Perplexity is the clearest of all: PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models".

User-triggered fetchers run when a person asks a question that requires opening your page. This is the category where robots.txt stops being a control. OpenAI: because these actions are initiated by a user, robots.txt rules may not apply. Perplexity: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Anthropic is the exception and documents Claude-User as controllable through robots.txt.

If you take one thing from this page: the operators built separate tokens so that "do not train on me" and "do not cite me" could be answered differently. A rule that blocks a whole vendor answers both at once, and almost nobody wants that.

How to write the rules

Opt out of training everywhere, stay fully citable:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Three mechanics that trip people up. Consecutive User-agent lines form one group sharing the rules that follow, so a stray blank line changes the meaning. The most specific matching group wins outright — an agent with its own group ignores * completely, which means a site-wide Disallow: / under * does not apply to a bot you named elsewhere. And within a group the longest matching path wins, with Allow beating Disallow on an exact tie.

Changes are not instant: OpenAI notes it can take around 24 hours from a robots.txt update for its search systems to adjust, and Perplexity says the same. If you want to see how your file resolves for a given agent and URL before you ship it, the robots.txt tester shows the deciding rule, and the AI crawler checker runs the whole named list against your domain at once.

Verifying a crawler is really who it claims

A user-agent string is self-declared. Anyone can send ClaudeBot in a header, and plenty of scrapers do — which matters both ways: if you allow a bot by name, you are allowing anyone who types that name.

Every major operator publishes the IP ranges its crawlers use as JSON, so verification is a lookup rather than a guess:

• OpenAI publishes separate files per agent — gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json.
• Anthropic publishes a single list at claude.com/crawling/bots.json, and warns that blocking those IPs is not a reliable opt-out because it also stops them reading your robots.txt.
• Perplexity publishes perplexitybot.json and perplexity-user.json.
• Apple documents reverse-DNS verification for Applebot.

The pattern for any allow-list worth having: match the user agent, then confirm the source IP is in the published range or that reverse DNS resolves to the operator's domain and forward-resolves back. Anything else is an honour system.

The mistakes we found in the wild

From reading 326 real files, in order of how often they cost the site something:

• Blocking PerplexityBot to prevent training. It does not train. 12.3% of top sites block it, giving up citations for nothing.
• Blocking Google-Extended in the belief it protects search. Google states it has no effect on Search inclusion or ranking; what it removes is Gemini grounding. The trade is the opposite of the one usually intended.
• Blocking the user-triggered fetchers. Two of the three are documented as possibly ignoring the rule. If it matters, enforce it at the edge; if it does not, the line only removes answers that would have linked you.
• Copying a block list from a blog post. Token names change, new ones appear — OAI-SearchBot and Claude-SearchBot did not exist when many of these files were written. A list from last year silently blocks the wrong things.
• Blocking nothing but shipping JavaScript-only pages. None of the major AI crawlers execute JavaScript, so an empty allow-all robots.txt still yields an empty page. We measured that separately in do AI crawlers render JavaScript: 6.4% of top sites have readable content only with JS.

Checking your own site

Three questions, in order. Does your robots.txt resolve the way you think for each named agent? Does your server return the content in raw HTML, or only after JavaScript? Does a CDN or WAF rule block a crawler that robots.txt allows — a mismatch invisible to anyone reading the file alone?

The first is the AI crawler checker. The second is the SEO audit API, which fetches a URL twice — once as a plain HTTP client with no JavaScript, once fully rendered — and returns both views plus the diff. The third needs a request from outside your network with the agent string set, which is what a scraping API does by default.

And if the answer to any of them is not what you expected, you are in good company: the gap between what these files say and what their owners meant was the single most common finding in our audit.