robots.txt tester

How the verdict is decided

A robots.txt file is not read top to bottom until something matches. Crawlers that follow RFC 9309 apply three rules in order, and this tester applies the same three:

1. One group wins. The crawler looks for the group whose User-agent names it most specifically. User-agent: * is only a fallback — if a group names the bot directly, the wildcard group is ignored completely, including its Disallow lines.
2. The longest matching rule wins. Not the first, not the last. Disallow: /private/ and Allow: /private/report.pdf can live in the same group: the second is longer, so that one file stays crawlable.
3. On a tie, Allow wins. When an Allow and a Disallow match with the same length, the URL is crawlable.

Two details catch people out. An empty Disallow: restricts nothing — it is the standard way of saying "everything is open", not "block everything". And * and $ are the only wildcards: * stands for any run of characters, $ anchors the end of the path, so /*.pdf$ matches /a/b.pdf but not /a/b.pdf.html.

Search crawlers and AI crawlers are different bots

Blocking an AI training crawler does not affect how your pages rank. They are separate user-agents run for separate purposes, and the distinction is the single most common misunderstanding we see in robots.txt files:

Google-Extended is the clearest example: it controls whether your content trains Gemini, and has no effect on Google Search ranking. Googlebot is what crawls for Search. Blocking one leaves the other untouched.

What robots.txt cannot do

It is a crawling instruction, not an access control and not an indexing control. A disallowed URL can still appear in search results if other sites link to it, because the crawler never fetches the page and therefore never sees a noindex. To keep a page out of an index, allow the crawl and use noindex; to keep it private, require authentication. And well-behaved crawlers obey the file — nothing enforces it on the ones that do not.