Why this matters
An AI assistant can only describe or cite your site if its crawler may read it. Many sites block these crawlers without knowing: a security plugin, a hosting default or a copied robots.txt can shut them all out.
Training crawlers and search crawlers are different
Most AI companies run more than one crawler. One collects pages that may be used to train models; another finds pages to show in answers; a third opens a page when a user asks for it. You can block training and still allow search, so your pages can be cited without being used for training. The table shows each one separately.
How to allow a crawler
Open your robots.txt and remove the Disallow rule that applies to it, or add a group for it with Allow: /. Then check again here. For the next step, publish an llms.txt with the free llms.txt generator and read how to get recommended by ChatGPT.
Questions people ask
Does blocking GPTBot remove me from ChatGPT?
Blocking GPTBot stops pages being collected for training. ChatGPT search uses OAI-SearchBot, and fetching a page for a user uses ChatGPT-User. Check all three if you want to appear in ChatGPT answers.
Does Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your pages are used for Gemini and related AI features. Google Search uses Googlebot.
Is robots.txt enough to keep AI crawlers out?
Reputable crawlers follow it, but it is a request, not a lock. Content you must keep private belongs behind a login.
How often should I check?
After any change to your site, hosting or security settings, because those can change robots.txt without you noticing.