Advertisement
Guide
About the Robots.txt Generator & Tester
robots.txt is a plain-text file at the root of a website that tells crawlers which parts of the site they may visit. Each group names one or more bots with User-agent and lists paths they are allowed or disallowed to fetch. Search engines read it before crawling, and so do most AI data collectors.
This generator builds a valid file from simple fields, and the tester shows exactly which rule decides whether a given bot may crawl a given URL.
robots.txt does not hide pages
Disallowing a URL stops well-behaved crawlers from fetching it, but the URL can still appear in search results if other sites link to it, and the file itself is public — anyone can read it. To keep a page out of search results, let it be crawled and add a noindex meta tag. To keep it private, put it behind a login.
Blocking AI crawlers
AI companies publish the user-agent names of the bots that collect training data, such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training) and CCBot (Common Crawl). Disallowing them leaves normal search crawling — Googlebot, Bingbot — untouched. Bots that fetch pages to answer AI search queries, such as OAI-SearchBot and PerplexityBot, are separate; block them only if you also want to stay out of AI search answers. Compliance is voluntary, so this stops reputable crawlers rather than every scraper.
How to use it
- 1Pick a preset or import your current robots.txt.
- 2Edit user-agent groups and their allow or disallow paths.
- 3Add your sitemap URL and, if you like, block AI crawlers.
- 4Test a few URLs, then copy or download the file to your site’s root.
Frequently asked questions
Where do I put robots.txt?
At the root of the host, so it is served at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
What is the difference between Allow and Disallow?
Disallow blocks paths that start with the given value; Allow re-opens part of a blocked area. When both match, the longer rule wins, and Allow wins a tie.
What do * and $ mean in robots.txt?
* matches any sequence of characters and $ anchors the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf.
Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay, though Bing, Yandex and some other crawlers honour it.
How do I block ChatGPT and other AI bots from training on my site?
Tick “Block AI training crawlers”. It adds a group for GPTBot, ClaudeBot, Google-Extended, CCBot and other training crawlers with Disallow: /. AI search bots like OAI-SearchBot are not included; add a group for them if you want those blocked too.
Will an empty Disallow block my site?
No. “Disallow:” with nothing after it allows everything. “Disallow: /” blocks the whole site.