AI Crawler Checker
Enter a website to see which AI crawlers its robots.txt allows or blocks, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended, and whether it has an llms.txt file. Then build your own AI rules below.
Build your AI crawler rules
Choose what to allow, then add these lines to your robots.txt (they work alongside your existing rules).
llms.txt generator
llms.txt is a proposed file that gives AI tools a short, clean guide to your site. Save the result as /llms.txt at the root of your website.
Three kinds of AI crawler
AI companies now send different bots for different jobs, and you can treat each one differently:
- AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) collect content that may be used to train AI models. Blocking them keeps your future content out of training data. It doesn’t remove anything already collected.
- AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so they can be shown and linked in AI answers. Block these and your site is less likely to be cited by ChatGPT search, Claude or Perplexity.
- User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page only when a person asks the AI about it. OpenAI and Perplexity both say these may not follow robots.txt, because a human started the request.
Many publishers block training but allow search, so they keep control of their content and still get cited in AI answers. The rule builder above makes that easy.
Does blocking Google-Extended remove me from Google?
No. Google-Extended is not a separate crawler; it is a token in robots.txt that controls whether Google may use your content for Gemini. Google Search, and AI Overviews in Search, use the normal Googlebot. If you block Googlebot, you disappear from Google Search, so never do that by accident.
How this checker reads robots.txt
It follows the official robots.txt standard (RFC 9309): a crawler obeys the group that names it, otherwise the User-agent: * group; the most specific (longest) matching rule wins, and Allow wins a tie. We test the homepage and a typical inner page for each bot. Small exceptions, such as a blocked /admin/ folder, are listed but don’t count as “blocked”. If a site’s robots.txt returns a server error, the standard says crawlers should treat the site as fully blocked.
robots.txt is a request, not a lock. Well-behaved crawlers follow it; badly behaved ones don’t. Firewalls and bot-protection services (such as Cloudflare’s AI bot blocking) can also stop AI crawlers, and that isn’t visible in robots.txt.
What is llms.txt?
llms.txt is a proposed standard: a short Markdown file at /llms.txt that points AI tools to your most useful pages. It is cheap to add, but no major AI company has confirmed that its systems rely on it, so treat it as optional. Your robots.txt rules and good content matter far more.
How to use the AI Crawler Checker
- Enter any website address, such as your own or a competitor’s.
- Read which AI crawlers are allowed or blocked for search, training and user requests.
- Use the rule builder to make your own AI robots.txt rules, and the generator for an llms.txt file.
Frequently asked questions
Should I block AI crawlers?
It depends on what you want. Blocking training bots keeps future content out of AI training. Blocking AI search bots also makes it less likely you’ll be cited in ChatGPT, Claude or Perplexity answers. Many sites block training and allow search.
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, which you control separately.
Why does it say a bot is allowed when my host blocks AI bots?
This tool reads robots.txt. Firewalls and bot-protection settings can block crawlers separately, and those rules aren’t public.