Should I Block AI Crawlers? New Data

Blocking the training crawlers is defensible. Blocking the search crawlers costs you citations and buys you nothing, because by their own operators' documentation they do not train on your content. We read the robots.txt of 326 of the web's top domains: 17.8% block GPTBot against 2.5% for Googlebot — but 12.3% also block PerplexityBot, which does not train at all.

What we measured

On 2 September 2026 we fetched /robots.txt for the top 800 apex domains of the Tranco research ranking — the same corpus as our no-JavaScript study, so the two datasets cross-reference. 362 domains answered; 326 returned a parseable robots.txt and form the sample. 36 served no robots.txt at all, which under the protocol means everything is permitted.

For each site we parsed the file into user-agent groups and resolved the verdict for the site root using the standard rules: the most specific matching group wins, longest matching path wins within it, and Allow beats Disallow on a tie. "Blocked" below always means blocked from the whole site, not from a subdirectory. The full per-domain results are published as a CSV so anyone can recheck the arithmetic.

The headline numbers

Crawler	What it does	Blocked	Named explicitly
CCBot	Common Crawl — public corpus	20.2%	19.0%
Bytespider	ByteDance — training	19.3%	16.9%
ClaudeBot	Anthropic — training	18.1%	20.2%
GPTBot	OpenAI — training	17.8%	19.9%
Google-Extended	Google — Gemini training and grounding	16.6%	19.0%
Applebot-Extended	Apple — training	15.0%	13.5%
Meta-ExternalAgent	Meta — training	14.7%	12.6%
Amazonbot	Amazon	13.2%	11.7%
PerplexityBot	Perplexity — search index only	12.3%	16.6%
ChatGPT-User	OpenAI — user-triggered fetch	9.5%	12.3%
Claude-SearchBot	Anthropic — search index	9.2%	9.5%
OAI-SearchBot	OpenAI — ChatGPT search index	8.3%	11.7%
Googlebot	Google Search	2.5%	11.3%
bingbot	Bing Search	3.4%	7.4%

Three things fall straight out of that table.

• AI crawlers are blocked about seven times more often than Googlebot. 17.8% against 2.5%.
• The blocking is deliberate, not incidental. Of the 58 sites that block GPTBot, only 8 do so because they block every unnamed crawler including Googlebot. 15.3% of all sites block GPTBot while explicitly letting Googlebot through — a choice about AI, not a general crawler wall.
• Training bots are blocked roughly twice as often as the search bots from the same company. GPTBot 17.8% versus OAI-SearchBot 8.3%. ClaudeBot 18.1% versus Claude-SearchBot 9.2%.

Publishers are drawing a line between training and citation

That last gap is the most interesting result in the dataset, because it is a policy position expressed in a config file.

OpenAI documents the choice explicitly: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to signal that the content should not be used for training. We measured how many actually take it. 9.5% of sites block GPTBot but allow OAI-SearchBot, and 8.9% block ClaudeBot but allow Claude-SearchBot. The stance is: do not learn from me, but do quote me.

That is a coherent and, in our view, correct position. The problem is how many sites get it wrong in the other direction.

Most of the blocking is aimed at the wrong crawler

Three findings that cost sites visibility for no benefit at all.

12.3% block PerplexityBot. Perplexity's own documentation states that PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models". Blocking it removes you from Perplexity's answers and protects nothing from training, because it was never training. This is close to a pure own goal.

16.6% block Google-Extended, and many will be surprised by what that does. Google-Extended controls Gemini training and grounding — the retrieval that pulls pages into an answer at prompt time. Google states it does not affect inclusion in Google Search and is not a ranking signal. So the token does exactly one thing you might want (no training) and one thing you probably do not (no Gemini citations), with no search downside either way. Whether that is a good trade is a real decision; our finding is that it is rarely being made as an informed one.

9.5% block ChatGPT-User, which may not obey the block. OpenAI's documentation says that because these fetches are initiated by a user, robots.txt rules may not apply. Perplexity is blunter: Perplexity-User "generally ignores robots.txt rules". A robots.txt line aimed at a user-triggered fetcher is a request, not a control. If you genuinely need to stop those fetches you need network-level enforcement, and if you do not, the line is only costing you answers.

Put together: a meaningful share of the top of the web has opted out of being cited by AI assistants while leaving training access unchanged or unenforceable. That is the opposite of what most of those site owners think they configured.

The bigger the site, the harder the block

Splitting by rank reproduces the pattern we found in the no-JavaScript study, where the largest sites were also the least readable.

Segment	Sites in sample	Block GPTBot	Name at least one AI crawler
Tranco top 100	26	34.6%	38.5%
Rank 101 and below	262	17.9%	29.0%

Twenty-six sites is a small subsample and we report it as indicative, not precise — but the direction matches what you would expect. Large publishers and platforms have licensing negotiations, legal departments and something to sell; the long tail mostly wants the traffic. The blocked names in our data are the ones you would guess: social platforms, portals and large media properties, several of which block everything unnamed while carving out Googlebot by name.

Blocking is not the only way to be invisible

Because this study uses the same corpus as our no-JavaScript audit, we can combine them. A crawler cannot read your site if you block it — and equally cannot read it if your content only exists after JavaScript runs, which none of the major AI crawlers execute.

Counting both failure modes together, 21.8% of top sites are unreadable to an AI crawler: either the robots.txt turns it away, or the HTML it receives is empty. Eight sites in the sample manage both at once. The second group is the interesting one, because unlike the first it is almost always an accident. Nobody decided to be invisible; a framework decided for them.

If you want to know which one applies to you, the AI crawler checker resolves your robots.txt against each named agent, and the SEO audit API fetches any URL twice — once as a plain HTTP client with no JavaScript, once fully rendered — and returns the diff.

So should you block AI crawlers?

The honest answer is that it depends on what you sell, and that the question is four questions. Ours are the numbers, not the decision.

1. If your content is the product — a paid archive, a licensed dataset, a subscription publication — blocking the training crawlers is rational, and 1 in 6 of the largest sites agrees with you. Block GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended.
2. If you sell something the content markets — almost every SaaS, agency, ecommerce and B2B site — the training question is close to irrelevant and the retrieval question is everything. Being absent from an assistant's answer is being absent from the shortlist. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot at minimum.
3. Decide the two separately. The operators built distinct tokens precisely so that you could, and a tenth of the sample already uses them that way. Blocking a whole vendor because you object to training is throwing away the citation with it.
4. Do not confuse robots.txt with enforcement. It is a published preference that well-behaved automated crawlers honour. User-triggered fetchers are documented as possibly ignoring it, and badly-behaved bots ignore it entirely. If the requirement is hard, it belongs at the edge.

And whichever way you go, check what the file actually says rather than what you meant it to say. In this sample the gap between the two was the single most common finding.

Methodology and limits

One fetch per apex domain on 2 September 2026, plain HTTP with no JavaScript, following redirects to the canonical host. Sample: 362 of the top 800 Tranco domains reachable, 326 with a parseable robots.txt. Excluded: DNS-infrastructure domains, hosts that returned an HTML error page in place of a text file, and non-200 responses — excluded rather than guessed at.

Limits worth stating. Robots.txt is a declaration, not evidence of behaviour: we measured what sites ask for, not what crawlers do. Root-level blocking undercounts partial restriction, so a site that allows the homepage and disallows /articles/ counts as allowed here. CDN and WAF rules are invisible to this method and can block a crawler that robots.txt permits. Agent naming changes: this snapshot uses the tokens the operators documented in September 2026. The top-100 cut is 26 sites and should be treated as directional. The raw per-domain CSV is published so you can apply your own definitions.