Glossary
What is robots.txt (AI crawlers)?
robots.txt is a file at a site's root that tells crawlers which paths they may request. AI companies operate their own named crawlers, such as GPTBot, ClaudeBot and PerplexityBot, which can be allowed or disallowed individually.
Blocking AI crawlers removes a site from the corpus those systems retrieve from, which reduces the chance of being cited. It is a trade-off between content control and AI visibility, and worth making deliberately.
Related terms
See how AI represents your brand today
Start measuring your first brand for free, or get a guided walkthrough with our team. Know where you stand before your competitors do.