robots.txt generator

What each choice actually does

A robots.txt file is a set of groups. Each group names one or more crawlers and lists what they may and may not fetch. The generator writes valid groups for you, but the decisions are yours, and they are not interchangeable:

Allowing search crawlers

Almost every public site should let search engines crawl everything except pages that waste crawl budget: internal search results, cart and checkout, faceted filters that multiply URLs, admin paths. Blocking those is about efficiency, not secrecy — anything genuinely private needs authentication, because robots.txt is a public file that anyone can read.

Blocking AI training crawlers

This is a business decision, not a technical one. Blocking GPTBot, ClaudeBot, CCBot or Google-Extended keeps your content out of future training runs, and changes nothing about how you rank. The trade-off runs the other way too: assistants that cite sources send traffic, and a site blocked from the index behind an assistant cannot be cited by it. Blocking training while allowing search and live-fetch bots is the middle path the second preset writes for you.

Crawl-delay

Leave it empty unless a crawler is measurably loading your server. Google ignores it entirely; Bing and Yandex honour it. A crawl-delay on a large site can mean pages go months without being recrawled.

Mistakes this generator avoids

• Blocking your CSS and JavaScript. Search engines render pages. A Disallow: /assets/ can make your site look broken to the crawler and cost you rankings.
• Using robots.txt to hide a page. A disallowed page can still be indexed from external links, because the crawler never fetches it and never sees your noindex.
• Forgetting the sitemap line. It is the cheapest discovery signal there is, and it belongs in robots.txt so any crawler finds it without being told.
• One file for every subdomain. Each host needs its own robots.txt; the file at example.com says nothing about shop.example.com.