Distribute
FREE TOOLNo sign-up

AI crawler checker: is your site blocking ChatGPT, Claude or Perplexity?

Enter your site and see, crawler by crawler, whether ChatGPT, Claude, Perplexity, Gemini and the others are allowed to read it. Free, no sign-up.

Why this matters

An AI assistant can only describe or cite your site if its crawler may read it. Many sites block these crawlers without knowing: a security plugin, a hosting default or a copied robots.txt can shut them all out.

Training crawlers and search crawlers are different

Most AI companies run more than one crawler. One collects pages that may be used to train models; another finds pages to show in answers; a third opens a page when a user asks for it. You can block training and still allow search, so your pages can be cited without being used for training. The table shows each one separately.

How to allow a crawler

Open your robots.txt and remove the Disallow rule that applies to it, or add a group for it with Allow: /. Then check again here. For the next step, publish an llms.txt with the free llms.txt generator and read how to get recommended by ChatGPT.

Questions people ask

Does blocking GPTBot remove me from ChatGPT?

Blocking GPTBot stops pages being collected for training. ChatGPT search uses OAI-SearchBot, and fetching a page for a user uses ChatGPT-User. Check all three if you want to appear in ChatGPT answers.

Does Google-Extended affect my Google rankings?

No. Google-Extended only controls whether your pages are used for Gemini and related AI features. Google Search uses Googlebot.

Is robots.txt enough to keep AI crawlers out?

Reputable crawlers follow it, but it is a request, not a lock. Content you must keep private belongs behind a login.

How often should I check?

After any change to your site, hosting or security settings, because those can change robots.txt without you noticing.

Find out where your distribution leaks

Type in your website. In about a minute you get a score out of 100 across search, AI answers, social, video, ads and links, with the fixes in order.

Free. No account. About 30 seconds.