AI crawler
Definition
An AI crawler is a bot that fetches web pages for an AI system, either to build training data, to power live answer retrieval, or to fetch a page a user asked about. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are AI crawlers.
Different crawlers do different jobs, and the difference matters: blocking a training crawler shapes what future models know, while blocking a retrieval crawler removes you from live answers today. Many sites block AI crawlers indiscriminately and then wonder why assistants describe them poorly.
Crawler traffic is also evidence. Which AI bots visit, what they read, and how often is an early signal of whether engines consider your site part of the reading list.
Related terms
