Distribute
FREE TOOLNo sign-up

Free robots.txt generator with AI crawler rules

Choose which AI crawlers may read your site, list the pages to keep out, and copy a ready robots.txt. Runs in your browser. Free, no sign-up.

Your robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /account/

Save it as robots.txt at the root of your site, so it opens at yoursite.com/robots.txt. Then test it with the AI crawler checker.

What robots.txt does

robots.txt is a text file at the root of your site that tells crawlers which pages they may visit. Search engines and the main AI companies follow it. It is a request, not a lock: never use it to hide private data, which needs a login instead.

AI search crawlers and AI training crawlers

AI companies run separate crawlers for different jobs. Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered ones) decide whether assistants can find and cite your pages. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages that may be used to train models. Many sites allow search and block training; if you want to be cited, keep the search crawlers allowed.

After you publish it

  1. Upload it so it opens at yoursite.com/robots.txt.
  2. Test it with the AI crawler checker.
  3. Add an llms.txt to tell AI assistants which pages matter most.

Questions people ask

Where does robots.txt go?

At the root of your domain, so it opens at yoursite.com/robots.txt. Each subdomain needs its own.

Does blocking GPTBot hide me from ChatGPT?

Not completely. GPTBot collects training data; ChatGPT search uses OAI-SearchBot and ChatGPT-User. Allow those to be cited in answers.

Can robots.txt remove a page from Google?

No. It stops crawling, but a page linked from elsewhere can still be indexed. Use a noindex tag to keep a page out of results.

Find out where your distribution leaks

Type in your website. In about a minute you get a score out of 100 across search, AI answers, social, video, ads and links, with the fixes in order.

Free. No account. About 30 seconds.