User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /account/ User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /account/ User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /account/
Save it as robots.txt at the root of your site, so it opens at yoursite.com/robots.txt. Then test it with the AI crawler checker.
What robots.txt does
robots.txt is a text file at the root of your site that tells crawlers which pages they may visit. Search engines and the main AI companies follow it. It is a request, not a lock: never use it to hide private data, which needs a login instead.
AI search crawlers and AI training crawlers
AI companies run separate crawlers for different jobs. Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered ones) decide whether assistants can find and cite your pages. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages that may be used to train models. Many sites allow search and block training; if you want to be cited, keep the search crawlers allowed.
After you publish it
- Upload it so it opens at yoursite.com/robots.txt.
- Test it with the AI crawler checker.
- Add an llms.txt to tell AI assistants which pages matter most.
Questions people ask
Where does robots.txt go?
At the root of your domain, so it opens at yoursite.com/robots.txt. Each subdomain needs its own.
Does blocking GPTBot hide me from ChatGPT?
Not completely. GPTBot collects training data; ChatGPT search uses OAI-SearchBot and ChatGPT-User. Allow those to be cited in answers.
Can robots.txt remove a page from Google?
No. It stops crawling, but a page linked from elsewhere can still be indexed. Use a noindex tag to keep a page out of results.