# AI Crawler User Agent List

> Every AI robots.txt token — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — what each does, what blocking costs, and measured block rates.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/AI Crawler User Agent List

# AI Crawler User Agent List

SEO dataSep 2, 2026·8 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [The complete list](/blog/ai-crawler-user-agent-list/#the-complete-list) [The three jobs, and why the distinction is the whole point](/blog/ai-crawler-user-agent-list/#the-three-jobs-and-why-the-distinction-is-the-whole-point) [How to write the rules](/blog/ai-crawler-user-agent-list/#how-to-write-the-rules) [Verifying a crawler is really who it claims](/blog/ai-crawler-user-agent-list/#verifying-a-crawler-is-really-who-it-claims) [The mistakes we found in the wild](/blog/ai-crawler-user-agent-list/#the-mistakes-we-found-in-the-wild) [Checking your own site](/blog/ai-crawler-user-agent-list/#checking-your-own-site)

There are three kinds of AI user agent, and confusing them is the most expensive mistake in robots.txt today. Training crawlers build models. Retrieval crawlers decide whether an assistant can cite you. User-triggered fetchers act on a person's request and are documented as possibly ignoring robots.txt entirely. Here is every token, what it does, and what blocking it costs.

## The complete list

Every entry below comes from the operator's own documentation. The last column is our own measurement: the share of 326 top Tranco domains that block that agent at the site root, audited on 2 September 2026 — the dataset behind [should I block AI crawlers](https://quanticdata.io/blog/should-i-block-ai-crawlers/).

| robots.txt token | Operator | Job | Blocking it means | Blocked by |
| --- | --- | --- | --- | --- |
| `GPTBot` | OpenAI | Training | Content should not be used to train OpenAI foundation models | 17.8% |
| `OAI-SearchBot` | OpenAI | Retrieval | Your site will not be shown in ChatGPT search answers | 8.3% |
| `ChatGPT-User` | OpenAI | User-triggered | May not apply — the fetch is initiated by a person | 9.5% |
| `OAI-AdsBot` | OpenAI | Ad landing-page checks | Only visits pages submitted as ads; not used for training | not measured |
| `ClaudeBot` | Anthropic | Training | Future material excluded from model training datasets | 18.1% |
| `Claude-SearchBot` | Anthropic | Retrieval | Content not indexed for search; reduced visibility in results | 9.2% |
| `Claude-User` | Anthropic | User-triggered | Content not retrieved in response to a user's question | 9.5% |
| `PerplexityBot` | Perplexity | Retrieval only | Removed from Perplexity results — **no training effect, it never trains** | 12.3% |
| `Perplexity-User` | Perplexity | User-triggered | Documented as generally ignoring robots.txt | 9.2% |
| `Google-Extended` | Google | Training + grounding control | No Gemini training and no grounding. **No effect on Google Search** | 16.6% |
| `Google-CloudVertexBot` | Google | Site-owner Vertex AI builds | Affects crawls site owners request for Vertex AI Agents only | not measured |
| `Applebot` | Apple | Search + training | Out of Spotlight, Siri and Safari suggestions | not measured |
| `Applebot-Extended` | Apple | Training control | No Apple foundation-model training. **Still appears in search results** | 15.0% |
| `CCBot` | Common Crawl | Public corpus | Out of a corpus many third parties reuse, including for training | 20.2% |
| `Bytespider` | ByteDance | Training | Excluded from ByteDance crawling | 19.3% |
| `Meta-ExternalAgent` | Meta | AI-related crawling | Excluded from Meta's AI crawling | 14.7% |
| `Amazonbot` | Amazon | Search and assistant | Excluded from Amazon crawling | 13.2% |

Two tokens in that table are not crawlers at all. `Google-Extended` has no HTTP user agent of its own — Google states the crawling is done with existing Google agents and the token is used purely as a control. `Applebot-Extended` likewise does not crawl; Apple describes it as only determining how data already collected by `Applebot` may be used. You cannot see either in your access logs, which is why people wrongly conclude they are not working.

## The three jobs, and why the distinction is the whole point

**Training crawlers** collect content that may go into a model's training data. Blocking them is a copyright-and-licensing decision. It has little bearing on whether an assistant can quote you today: when an assistant cites a source with a link, it is almost always because a retrieval crawler indexed that page, not because the content was in a training set.

**Retrieval crawlers** build the index an assistant searches at answer time. These are the ones that decide whether you get cited. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot prevents indexing and may reduce visibility. Perplexity is the clearest of all: PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models".

**User-triggered fetchers** run when a person asks a question that requires opening your page. This is the category where robots.txt stops being a control. OpenAI: because these actions are initiated by a user, robots.txt rules may not apply. Perplexity: since a user requested the fetch, this fetcher generally ignores robots.txt rules. Anthropic is the exception and documents Claude-User as controllable through robots.txt.

If you take one thing from this page: the operators built separate tokens so that "do not train on me" and "do not cite me" could be answered differently. A rule that blocks a whole vendor answers both at once, and almost nobody wants that.

## How to write the rules

Opt out of training everywhere, stay fully citable:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /
```

Three mechanics that trip people up. Consecutive `User-agent` lines form **one** group sharing the rules that follow, so a stray blank line changes the meaning. The most specific matching group wins outright — an agent with its own group ignores `*` completely, which means a site-wide `Disallow: /` under `*` does not apply to a bot you named elsewhere. And within a group the longest matching path wins, with `Allow` beating `Disallow` on an exact tie.

Changes are not instant: OpenAI notes it can take around 24 hours from a robots.txt update for its search systems to adjust, and Perplexity says the same. If you want to see how your file resolves for a given agent and URL before you ship it, the [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/) shows the deciding rule, and the [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/) runs the whole named list against your domain at once.

## Verifying a crawler is really who it claims

A user-agent string is self-declared. Anyone can send `ClaudeBot` in a header, and plenty of scrapers do — which matters both ways: if you allow a bot by name, you are allowing anyone who types that name.

Every major operator publishes the IP ranges its crawlers use as JSON, so verification is a lookup rather than a guess:

- OpenAI publishes separate files per agent — `gptbot.json`, `searchbot.json`, `chatgpt-user.json`, `adsbot.json`.

- Anthropic publishes a single list at `claude.com/crawling/bots.json`, and warns that blocking those IPs is not a reliable opt-out because it also stops them reading your robots.txt.

- Perplexity publishes `perplexitybot.json` and `perplexity-user.json`.

- Apple documents reverse-DNS verification for Applebot.

The pattern for any allow-list worth having: match the user agent, then confirm the source IP is in the published range or that reverse DNS resolves to the operator's domain and forward-resolves back. Anything else is an honour system.

## The mistakes we found in the wild

From reading 326 real files, in order of how often they cost the site something:

- **Blocking PerplexityBot to prevent training.** It does not train. 12.3% of top sites block it, giving up citations for nothing.

- **Blocking Google-Extended in the belief it protects search.** Google states it has no effect on Search inclusion or ranking; what it removes is Gemini grounding. The trade is the opposite of the one usually intended.

- **Blocking the user-triggered fetchers.** Two of the three are documented as possibly ignoring the rule. If it matters, enforce it at the edge; if it does not, the line only removes answers that would have linked you.

- **Copying a block list from a blog post.** Token names change, new ones appear — `OAI-SearchBot` and `Claude-SearchBot` did not exist when many of these files were written. A list from last year silently blocks the wrong things.

- **Blocking nothing but shipping JavaScript-only pages.** None of the major AI crawlers execute JavaScript, so an empty allow-all robots.txt still yields an empty page. We measured that separately in [do AI crawlers render JavaScript](https://quanticdata.io/blog/do-ai-crawlers-render-javascript/): 6.4% of top sites have readable content only with JS.

## Checking your own site

Three questions, in order. Does your robots.txt resolve the way you think for each named agent? Does your server return the content in raw HTML, or only after JavaScript? Does a CDN or WAF rule block a crawler that robots.txt allows — a mismatch invisible to anyone reading the file alone?

The first is the [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/). The second is the [SEO audit API](https://quanticdata.io/seo-audit/), which fetches a URL twice — once as a plain HTTP client with no JavaScript, once fully rendered — and returns both views plus the diff. The third needs a request from outside your network with the agent string set, which is what a [scraping API](https://quanticdata.io/web-scraping-api/) does by default.

And if the answer to any of them is not what you expected, you are in good company: the gap between what these files say and what their owners meant was the single most common finding in our audit.

### Sources & further reading

- [OpenAI — Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

- [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

- [Perplexity — Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

- [Google — List of Google’s common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

- [Apple — About Applebot and Applebot-Extended](https://support.apple.com/en-us/119829)

- [Google — How Google interprets the robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

- [Common Crawl — CCBot](https://commoncrawl.org/ccbot)

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

## FAQ

Quick answers on ai crawler user agent list.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What are the AI crawler user agents?

The main robots.txt tokens are GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Google-Extended and Google-CloudVertexBot (Google); Applebot and Applebot-Extended (Apple); plus CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta) and Amazonbot (Amazon).

### Which AI crawlers actually cite my site?

The retrieval crawlers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, and PerplexityBot for Perplexity. These build the index an assistant searches at answer time. The training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot — do not affect whether you can be cited today.

### Does Google-Extended affect Google Search rankings?

No. Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal. It is a control token with no HTTP user agent of its own, governing whether crawled content may be used for Gemini training and for grounding — supplying content to the model at prompt time.

### Do AI crawlers ignore robots.txt?

The automated crawlers state that they honour it and publish IP ranges so you can verify them. The user-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiated the request, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic documents Claude-User as controllable through robots.txt.

### What is the difference between Applebot and Applebot-Extended?

Applebot is the actual crawler powering Spotlight, Siri and Safari, and its data may also train Apple foundation models. Applebot-Extended does not crawl anything — it is a control token that opts your content out of that training. Apple states that pages disallowing Applebot-Extended can still appear in search results.

### How do I verify a request is really from GPTBot or ClaudeBot?

A user-agent string is self-declared, so check the source IP against the operator’s published ranges: OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json; Anthropic publishes claude.com/crawling/bots.json; Perplexity publishes perplexitybot.json and perplexity-user.json. Anthropic also warns that blocking its IPs is not a reliable opt-out, because it prevents them reading your robots.txt.

### How many sites block each AI crawler?

In our audit of 326 top Tranco domains on 2 September 2026: CCBot 20.2%, Bytespider 19.3%, ClaudeBot 18.1%, GPTBot 17.8%, Google-Extended 16.6%, Applebot-Extended 15.0%, Meta-ExternalAgent 14.7%, Amazonbot 13.2%, PerplexityBot 12.3%, ChatGPT-User 9.5%, Claude-SearchBot 9.2%, OAI-SearchBot 8.3% — against 2.5% for Googlebot.

## Check what your robots.txt actually says

The free AI crawler checker resolves your file against every named agent and shows the deciding rule; the SEO audit API adds the other half of the picture by fetching any URL twice — plain HTTP with no JavaScript, and fully rendered — at $0.0012 per URL. Failed requests are never billed, and every account gets $2 of free usage a month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore SEO Audit API](https://quanticdata.io/seo-audit/)

## Related reading

[SEO data Should I Block AI Crawlers? New Data We parsed the robots.txt of 326 of the web’s top domains. AI crawlers are blocked seven times more often than Googlebot, training bots twice as often as the search bots from the same company — and a large share of the blocking lands on crawlers that would have cited the site and never trained on it. Read →](https://quanticdata.io/blog/should-i-block-ai-crawlers/) [SEO data How Many Sites Use llms.txt? New Data 64 of 355 top domains publish a real llms.txt. Count by status code instead and you get 39.7%, because 77 sites answer 200 with their homepage. Includes what the real files contain, and the cross-tab nobody has run: publishers of llms.txt block AI crawlers four to seven times less, and none of them blocks a retrieval crawler. Read →](https://quanticdata.io/blog/how-many-sites-use-llms-txt/) [SEO data llms.txt vs robots.txt robots.txt answers whether an agent may fetch a page and is honoured by every major AI operator. llms.txt answers what is worth reading and is documented as read by none of them. The formats, the resolution rules, what neither file can do, and measured adoption for both across the same 355 domains. Read →](https://quanticdata.io/blog/llms-txt-vs-robots-txt/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/ai-crawler-user-agent-list/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt