Question 1

Should I block AI crawlers in robots.txt?

Accepted Answer

That depends on your goals. Blocking training crawlers (CCBot) is common and has no impact on AI search visibility. Blocking inference crawlers (GPTBot, ClaudeBot, PerplexityBot) means those AI engines cannot read your pages when generating answers, which reduces your chance of being cited. Blocking Googlebot-related crawlers like Google-Extended affects whether Google's AI overview can reference your content.

Question 2

What is the difference between a training crawler and an inference crawler?

Accepted Answer

Training crawlers collect data to train AI models (CCBot for Common Crawl, used by many LLMs). Inference crawlers fetch pages at query time to produce answers (GPTBot for ChatGPT, PerplexityBot for Perplexity, ClaudeBot for Claude). Blocking training crawlers prevents future model training. Blocking inference crawlers prevents the AI from citing your page in live answers.

Question 3

Does a User-agent: * block stop all AI crawlers?

Accepted Answer

Yes, if the Disallow rule covers the paths you care about. User-agent: * with Disallow: / blocks all crawlers, including all AI bots, from reading your entire site. Some sites use this accidentally, or inherit it from old WordPress defaults.

Question 4

Is this robots.txt check the same as what crawlers actually do?

Accepted Answer

This tool reads the robots.txt file and reports the stated rules. In practice, compliance is voluntary. Most major AI companies (OpenAI, Anthropic, Google) say they respect robots.txt, but nothing enforces it. The tool tells you what the file says, not whether every crawler obeys it.