# Should I Block AI Crawlers? New Data

> We read robots.txt on 326 top domains: 17.8% block GPTBot vs 2.5% Googlebot. But 12.3% block PerplexityBot, which by its own docs never trains on you.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/Should I Block AI Crawlers? New Data

# Should I Block AI Crawlers? New Data

SEO dataSep 2, 2026·9 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [What we measured](/blog/should-i-block-ai-crawlers/#what-we-measured) [The headline numbers](/blog/should-i-block-ai-crawlers/#the-headline-numbers) [Publishers are drawing a line between training and citation](/blog/should-i-block-ai-crawlers/#publishers-are-drawing-a-line-between-training-and-citation) [Most of the blocking is aimed at the wrong crawler](/blog/should-i-block-ai-crawlers/#most-of-the-blocking-is-aimed-at-the-wrong-crawler) [The bigger the site, the harder the block](/blog/should-i-block-ai-crawlers/#the-bigger-the-site-the-harder-the-block) [Blocking is not the only way to be invisible](/blog/should-i-block-ai-crawlers/#blocking-is-not-the-only-way-to-be-invisible) [So should you block AI crawlers?](/blog/should-i-block-ai-crawlers/#so-should-you-block-ai-crawlers) [Methodology and limits](/blog/should-i-block-ai-crawlers/#methodology-and-limits)

Blocking the training crawlers is defensible. Blocking the search crawlers costs you citations and buys you nothing, because by their own operators' documentation they do not train on your content. We read the robots.txt of 326 of the web's top domains: 17.8% block GPTBot against 2.5% for Googlebot — but 12.3% also block PerplexityBot, which does not train at all.

## What we measured

On 2 September 2026 we fetched `/robots.txt` for the top 800 apex domains of the [Tranco research ranking](https://tranco-list.eu/) — the same corpus as our [no-JavaScript study](https://quanticdata.io/blog/do-ai-crawlers-render-javascript/), so the two datasets cross-reference. 362 domains answered; 326 returned a parseable robots.txt and form the sample. 36 served no robots.txt at all, which under the protocol means everything is permitted.

For each site we parsed the file into user-agent groups and resolved the verdict for the site root using the standard rules: the most specific matching group wins, longest matching path wins within it, and `Allow` beats `Disallow` on a tie. "Blocked" below always means *blocked from the whole site*, not from a subdirectory. The full per-domain results are published as [a CSV](https://quanticdata.io/blog/should-i-block-ai-crawlers/ai-crawler-robots-study.csv?v=20260902) so anyone can recheck the arithmetic.

## The headline numbers

| Crawler | What it does | Blocked | Named explicitly |
| --- | --- | --- | --- |
| CCBot | Common Crawl — public corpus | **20.2%** | 19.0% |
| Bytespider | ByteDance — training | **19.3%** | 16.9% |
| ClaudeBot | Anthropic — training | **18.1%** | 20.2% |
| GPTBot | OpenAI — training | **17.8%** | 19.9% |
| Google-Extended | Google — Gemini training and grounding | **16.6%** | 19.0% |
| Applebot-Extended | Apple — training | **15.0%** | 13.5% |
| Meta-ExternalAgent | Meta — training | **14.7%** | 12.6% |
| Amazonbot | Amazon | **13.2%** | 11.7% |
| PerplexityBot | Perplexity — *search index only* | **12.3%** | 16.6% |
| ChatGPT-User | OpenAI — user-triggered fetch | **9.5%** | 12.3% |
| Claude-SearchBot | Anthropic — search index | **9.2%** | 9.5% |
| OAI-SearchBot | OpenAI — ChatGPT search index | **8.3%** | 11.7% |
| Googlebot | Google Search | **2.5%** | 11.3% |
| bingbot | Bing Search | **3.4%** | 7.4% |

Three things fall straight out of that table.

- **AI crawlers are blocked about seven times more often than Googlebot.** 17.8% against 2.5%.

- **The blocking is deliberate, not incidental.** Of the 58 sites that block GPTBot, only 8 do so because they block every unnamed crawler including Googlebot. **15.3% of all sites block GPTBot while explicitly letting Googlebot through** — a choice about AI, not a general crawler wall.

- **Training bots are blocked roughly twice as often as the search bots from the same company.** GPTBot 17.8% versus OAI-SearchBot 8.3%. ClaudeBot 18.1% versus Claude-SearchBot 9.2%.

## Publishers are drawing a line between training and citation

That last gap is the most interesting result in the dataset, because it is a policy position expressed in a config file.

OpenAI documents the choice explicitly: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to signal that the content should not be used for training. We measured how many actually take it. **9.5% of sites block GPTBot but allow OAI-SearchBot**, and **8.9% block ClaudeBot but allow Claude-SearchBot**. The stance is: do not learn from me, but do quote me.

That is a coherent and, in our view, correct position. The problem is how many sites get it wrong in the other direction.

## Most of the blocking is aimed at the wrong crawler

Three findings that cost sites visibility for no benefit at all.

**12.3% block PerplexityBot.** Perplexity's own documentation states that PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models". Blocking it removes you from Perplexity's answers and protects nothing from training, because it was never training. This is close to a pure own goal.

**16.6% block Google-Extended, and many will be surprised by what that does.** Google-Extended controls Gemini training *and grounding* — the retrieval that pulls pages into an answer at prompt time. Google states it does not affect inclusion in Google Search and is not a ranking signal. So the token does exactly one thing you might want (no training) and one thing you probably do not (no Gemini citations), with no search downside either way. Whether that is a good trade is a real decision; our finding is that it is rarely being made as an informed one.

**9.5% block ChatGPT-User, which may not obey the block.** OpenAI's documentation says that because these fetches are initiated by a user, robots.txt rules may not apply. Perplexity is blunter: Perplexity-User "generally ignores robots.txt rules". A robots.txt line aimed at a user-triggered fetcher is a request, not a control. If you genuinely need to stop those fetches you need network-level enforcement, and if you do not, the line is only costing you answers.

Put together: a meaningful share of the top of the web has opted out of being cited by AI assistants while leaving training access unchanged or unenforceable. That is the opposite of what most of those site owners think they configured.

## The bigger the site, the harder the block

Splitting by rank reproduces the pattern we found in the no-JavaScript study, where the largest sites were also the least readable.

| Segment | Sites in sample | Block GPTBot | Name at least one AI crawler |
| --- | --- | --- | --- |
| Tranco top 100 | 26 | **34.6%** | 38.5% |
| Rank 101 and below | 262 | **17.9%** | 29.0% |

Twenty-six sites is a small subsample and we report it as indicative, not precise — but the direction matches what you would expect. Large publishers and platforms have licensing negotiations, legal departments and something to sell; the long tail mostly wants the traffic. The blocked names in our data are the ones you would guess: social platforms, portals and large media properties, several of which block everything unnamed while carving out Googlebot by name.

## Blocking is not the only way to be invisible

Because this study uses the same corpus as our no-JavaScript audit, we can combine them. A crawler cannot read your site if you block it — and equally cannot read it if your content only exists after JavaScript runs, which none of the major AI crawlers execute.

Counting both failure modes together, **21.8% of top sites are unreadable to an AI crawler**: either the robots.txt turns it away, or the HTML it receives is empty. Eight sites in the sample manage both at once. The second group is the interesting one, because unlike the first it is almost always an accident. Nobody decided to be invisible; a framework decided for them.

If you want to know which one applies to you, the [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/) resolves your robots.txt against each named agent, and the [SEO audit API](https://quanticdata.io/seo-audit/) fetches any URL twice — once as a plain HTTP client with no JavaScript, once fully rendered — and returns the diff.

## So should you block AI crawlers?

The honest answer is that it depends on what you sell, and that the question is four questions. Ours are the numbers, not the decision.

1. **If your content is the product** — a paid archive, a licensed dataset, a subscription publication — blocking the training crawlers is rational, and 1 in 6 of the largest sites agrees with you. Block GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended.

2. **If you sell something the content markets** — almost every SaaS, agency, ecommerce and B2B site — the training question is close to irrelevant and the retrieval question is everything. Being absent from an assistant's answer is being absent from the shortlist. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot at minimum.

3. **Decide the two separately.** The operators built distinct tokens precisely so that you could, and a tenth of the sample already uses them that way. Blocking a whole vendor because you object to training is throwing away the citation with it.

4. **Do not confuse robots.txt with enforcement.** It is a published preference that well-behaved automated crawlers honour. User-triggered fetchers are documented as possibly ignoring it, and badly-behaved bots ignore it entirely. If the requirement is hard, it belongs at the edge.

And whichever way you go, check what the file actually says rather than what you meant it to say. In this sample the gap between the two was the single most common finding.

## Methodology and limits

One fetch per apex domain on 2 September 2026, plain HTTP with no JavaScript, following redirects to the canonical host. Sample: 362 of the top 800 Tranco domains reachable, 326 with a parseable robots.txt. Excluded: DNS-infrastructure domains, hosts that returned an HTML error page in place of a text file, and non-200 responses — excluded rather than guessed at.

Limits worth stating. Robots.txt is a declaration, not evidence of behaviour: we measured what sites ask for, not what crawlers do. Root-level blocking undercounts partial restriction, so a site that allows the homepage and disallows `/articles/` counts as allowed here. CDN and WAF rules are invisible to this method and can block a crawler that robots.txt permits. Agent naming changes: this snapshot uses the tokens the operators documented in September 2026. The top-100 cut is 26 sites and should be treated as directional. The [raw per-domain CSV](https://quanticdata.io/blog/should-i-block-ai-crawlers/ai-crawler-robots-study.csv?v=20260902) is published so you can apply your own definitions.

### Sources & further reading

- [OpenAI — Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)](https://developers.openai.com/api/docs/bots)

- [Anthropic — Does Anthropic crawl data from the web (ClaudeBot, Claude-User, Claude-SearchBot)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

- [Perplexity — Perplexity Crawlers (PerplexityBot, Perplexity-User)](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

- [Google — List of Google’s common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

- [Google — Robots.txt specification and order of precedence](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

- [Tranco — a research-oriented top sites ranking](https://tranco-list.eu/)

- [Common Crawl — CCBot and the public web corpus](https://commoncrawl.org/ccbot)

## FAQ

Quick answers on should i block ai crawlers.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### What percentage of websites block AI crawlers?

In our September 2026 audit of 326 top Tranco domains, 17.8% blocked GPTBot at the site root, 18.1% blocked ClaudeBot and 20.2% blocked CCBot — against 2.5% for Googlebot. 23.6% blocked at least one major AI crawler. 36 of the 362 reachable domains served no robots.txt at all, which permits everything.

### Should I block AI crawlers?

Split the decision. If your content is the product you sell, blocking the training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended — is rational. If your content markets something else you sell, blocking costs you citations in AI answers and protects revenue you do not have. Either way, allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot exist to surface and link you, not to train.

### Does blocking PerplexityBot stop AI training on my content?

No. Perplexity documents that PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models. Blocking it removes you from Perplexity answers and prevents no training whatsoever. 12.3% of the top sites we measured block it anyway.

### Does blocking Google-Extended hurt my Google Search ranking?

No. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal. What it does control is Gemini training and grounding — the retrieval that pulls pages into an AI answer at prompt time. So blocking it costs you Gemini citations without costing you search rankings. 16.6% of the sites we measured block it.

### Do AI crawlers actually obey robots.txt?

The automated crawlers from the major vendors publish IP ranges and state that they honour it. The user-triggered fetchers are different: OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt. Robots.txt is a published preference, not enforcement — for hard requirements use network-level controls.

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI’s foundation models; disallowing it signals your content should not be used in training. OAI-SearchBot surfaces sites in ChatGPT’s search features, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers. The settings are independent, which is why 9.5% of the sites we measured block one and allow the other.

### How can I check which AI crawlers my site blocks?

Read your own robots.txt and resolve it per agent rather than by eye — the most specific user-agent group wins, and the longest matching path wins inside it, which is where most surprises come from. QuanticData’s free AI crawler checker does this for the named agents, and the SEO audit API additionally shows what a crawler sees without JavaScript.

## See your site the way an AI crawler does

The free AI crawler checker resolves your robots.txt per agent, and the SEO audit API fetches any URL twice — as a plain HTTP bot with no JavaScript and fully rendered — returning both views and the diff at $0.0012 per URL. Failed requests are never billed, and every account gets $2 of free usage a month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore SEO Audit API](https://quanticdata.io/seo-audit/)

## Related reading

[SEO data AI Crawler User Agent List A reference table of every AI user agent: which operator runs it, whether its job is training, retrieval or a user-triggered fetch, what blocking it actually costs you, and the share of 326 top domains that block it today. Plus the robots.txt mechanics that make these rules mean something other than intended. Read →](https://quanticdata.io/blog/ai-crawler-user-agent-list/) [SEO data How Many Sites Use llms.txt? New Data 64 of 355 top domains publish a real llms.txt. Count by status code instead and you get 39.7%, because 77 sites answer 200 with their homepage. Includes what the real files contain, and the cross-tab nobody has run: publishers of llms.txt block AI crawlers four to seven times less, and none of them blocks a retrieval crawler. Read →](https://quanticdata.io/blog/how-many-sites-use-llms-txt/) [SEO data llms.txt vs robots.txt robots.txt answers whether an agent may fetch a page and is honoured by every major AI operator. llms.txt answers what is worth reading and is documented as read by none of them. The formats, the resolution rules, what neither file can do, and measured adoption for both across the same 355 domains. Read →](https://quanticdata.io/blog/llms-txt-vs-robots-txt/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/should-i-block-ai-crawlers/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt