# robots.txt Tester — Check Any URL, Any Crawler

> Paste a robots.txt, test any URLs against Googlebot, GPTBot or ClaudeBot, and see the exact rule that allowed or blocked each one. Free, no upload.

[Home](https://quanticdata.io/)/[Free tools](https://quanticdata.io/tools/)/robots.txt tester

# robots.txt tester

Paste a robots.txt file and a list of URLs, pick a crawler, and get a verdict for each URL with the exact line that decided it. It covers search bots and AI crawlers, applies RFC 9309 precedence, and runs entirely in your browser — nothing you paste is uploaded.

Free, no sign-up, nothing leaves this page.

robots.txt content User-agent: * Disallow: /admin/ Disallow: /cart Allow: /admin/public-policy.html User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml

Crawler GooglebotGooglebot-ImageBingbot DuckDuckBotYandexBotApplebot GPTBotOAI-SearchBotChatGPT-User ClaudeBotPerplexityBotGoogle-Extended CCBotBytespiderApplebot-Extended meta-externalagent AhrefsBotSemrushBot* URLs or paths to test, one per line /admin/settings /admin/public-policy.html /cart?id=3 /blog/how-we-built-it https://example.com/pricing

## How the verdict is decided

A robots.txt file is not read top to bottom until something matches. Crawlers that follow [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) apply three rules in order, and this tester applies the same three:

1. **One group wins.** The crawler looks for the group whose `User-agent` names it most specifically. `User-agent: *` is only a fallback — if a group names the bot directly, the wildcard group is ignored completely, including its `Disallow` lines.

2. **The longest matching rule wins.** Not the first, not the last. `Disallow: /private/` and `Allow: /private/report.pdf` can live in the same group: the second is longer, so that one file stays crawlable.

3. **On a tie, Allow wins.** When an Allow and a Disallow match with the same length, the URL is crawlable.

Two details catch people out. An empty `Disallow:` restricts nothing — it is the standard way of saying "everything is open", not "block everything". And `*` and `$` are the only wildcards: `*` stands for any run of characters, `$` anchors the end of the path, so `/*.pdf$` matches `/a/b.pdf` but not `/a/b.pdf.html`.

## Search crawlers and AI crawlers are different bots

Blocking an AI training crawler does not affect how your pages rank. They are separate user-agents run for separate purposes, and the distinction is the single most common misunderstanding we see in robots.txt files:

| User-agent | What it does |
| --- | --- |
| `GPTBot` | OpenAI, training |
| `OAI-SearchBot` | OpenAI, search inside ChatGPT |
| `ChatGPT-User` | OpenAI, live fetch when a user asks |
| `ClaudeBot` | Anthropic, training |
| `PerplexityBot` | Perplexity, answers with citations |
| `Google-Extended` | Google, Gemini training only |
| `CCBot` | Common Crawl, the open training dataset |
| `Bytespider` | ByteDance, training |
| `Applebot-Extended` | Apple Intelligence training |
| `meta-externalagent` | Meta, training |

`Google-Extended` is the clearest example: it controls whether your content trains Gemini, and has no effect on Google Search ranking. `Googlebot` is what crawls for Search. Blocking one leaves the other untouched.

## What robots.txt cannot do

It is a crawling instruction, not an access control and not an indexing control. A disallowed URL can still appear in search results if other sites link to it, because the crawler never fetches the page and therefore never sees a `noindex`. To keep a page out of an index, allow the crawl and use `noindex`; to keep it private, require authentication. And well-behaved crawlers obey the file — nothing enforces it on the ones that do not.

## robots.txt questions people actually ask

### How do I check if a URL is blocked by robots.txt?

Paste the robots.txt file into the box above, add the URL, and pick the crawler you care about. The tester applies RFC 9309 precedence and shows the exact line that produced the verdict, so you can see *why* a URL is allowed or blocked rather than just that it is.

### Does blocking GPTBot remove my site from Google?

No. GPTBot is OpenAI’s training crawler and has nothing to do with Google Search. Googlebot crawls for Search, Google-Extended controls Gemini training. Blocking one does not affect the other.

### What does an empty Disallow mean?

A `Disallow:` line with no value restricts nothing — it is the standard way to say “the whole site is open”. To block everything you need `Disallow: /` with the slash.

### Which rule wins when Allow and Disallow both match?

The longer rule wins, measured in characters of the path. If both are the same length, Allow wins. That is why `Allow: /private/report.pdf` beats `Disallow: /private/` for that one file.

### Does robots.txt keep a page out of search results?

Not reliably. A blocked URL can still be listed if other pages link to it, because the crawler cannot fetch the page to see a noindex tag. Allow the crawl and add `noindex` if you want a page kept out of the index.

### Is my robots.txt uploaded anywhere by this tool?

No. The parser and the matcher are JavaScript running in your browser; the content you paste never leaves the page. That is also why the tool works on files that are not published yet.

### Where does robots.txt have to live?

At the root of the host it applies to — `https://example.com/robots.txt`. A file at a subpath is ignored, and subdomains need their own file.

## Related

[robots.txt generator Build the file, with AI-crawler presets](https://quanticdata.io/tools/robots-txt-generator/) [SEO audit API Audit crawlability across a whole site](https://quanticdata.io/seo-audit/) [Crawl & map API Discover every URL a site exposes](https://quanticdata.io/crawl-map/) [Web scraping API Fetch pages as clean Markdown](https://quanticdata.io/web-scraping-api/)

## Check crawlability across a whole site, not one URL at a time

This page tests the rules by hand. The SEO audit and crawl APIs do it across every URL a site exposes, and return the result as JSON or Markdown.

[Get my free API key](https://app.quanticdata.io/register)

Free key, $2 of usage credit every month, no credit card.

---

Source: https://quanticdata.io/tools/robots-txt-tester/ · Site index for AI: https://quanticdata.io/llms.txt