See which AI tools your robots.txt allows or blocks, and how to fix it.
Reading your robots.txt…
What Is robots.txt?
robots.txt is a small text file at the root of your website, for example yourbusiness.com/robots.txt. It tells crawlers, the automated visitors that search engines and AI companies send, which parts of your site they may read. One wrong line can stop ChatGPT or Perplexity from ever reading your pages, and many business owners never know it is there.
The Three Types of AI Crawlers
AI search and answers
Crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot find your pages so AI tools can quote and recommend you. Keep these allowed if you want to be found.
Live visits
ChatGPT-User, Claude-User and Perplexity-User visit your page when someone asks an AI assistant about it, for example by pasting your link into a chat.
AI training
GPTBot, ClaudeBot, Google-Extended, CCBot and others collect content to train AI models. Blocking them is a business choice and does not stop you appearing in AI search.
AI Crawler Checker: Frequently Asked Questions
Should I block AI crawlers?
If you want customers to find you through ChatGPT, Perplexity, Google AI Overviews or Copilot, keep the search crawlers allowed. You can block training crawlers separately if you prefer, without losing visibility.
Does blocking GPTBot remove me from ChatGPT?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot does not block OAI-SearchBot, so you can still appear in ChatGPT search results.
My robots.txt allows everything. Am I safe?
Not always. A firewall or security setting, such as Cloudflare’s AI bot blocking, can still turn AI crawlers away. The AI Visibility Checker tests that too.
How do I edit robots.txt on WordPress?
SEO plugins such as Rank Math (General Settings, Edit robots.txt) and Yoast (Tools, File editor) let you edit it from your dashboard without touching code.