Glossary
What is robots.txt (AI crawlers)?
robots.txt is a file at a site's root that tells crawlers which paths they may request. AI companies operate their own named crawlers, such as GPTBot, ClaudeBot and PerplexityBot, which can be allowed or disallowed individually.
Blocking AI crawlers removes a site from the corpus those systems retrieve from, which reduces the chance of being cited. It is a trade-off between content control and AI visibility, and worth making deliberately.
Related terms