Free tools · Directory
The AI bot directory.
Every major AI crawler in one place: who operates it, what it feeds, the user agent it publishes, and the robots.txt lines that allow or block it. Grouped by what each bot is actually doing on your site.
Which of these can actually read your site?
Check my siteLive answer fetchers
Fetches your pages in real time while an AI model answers a question. The strongest signal AI is reading you live.
| Bot | Operator | What it does | robots.txt |
|---|---|---|---|
| ChatGPT-User | OpenAI | Fetches a page during a ChatGPT session when the model needs it. Documented by OpenAI as user-triggered. | Honoured |
| Claude-User | Anthropic | Fetches a page during a Claude session when the model needs it. Documented by Anthropic as user-triggered. | Honoured |
| Perplexity-User | Perplexity | Fetches a page during a Perplexity session when the model needs it. Documented by Perplexity as user-triggered. | Honoured |
| DuckAssistBot | DuckDuckGo | Fetches a page when a DuckDuckGo session calls for it. Documented by DuckDuckGo as user-triggered. | Honoured |
| MistralAI-User | Mistral | Fetches a page during a Le Chat session when the model needs it. Documented by Mistral as user-triggered. | Honoured |
AI search crawlers
Retrieves and verifies pages so an AI model can cite them as sources.
| Bot | Operator | What it does | robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Crawls to build the index behind ChatGPT Search results. Documented by OpenAI as a search crawler. | Honoured |
| Claude-SearchBot | Anthropic | Crawls to build the index behind Claude’s web search. Documented by Anthropic as a search crawler. | Honoured |
| PerplexityBot | Perplexity | Crawls to build the index Perplexity cites from. Documented by Perplexity as a search crawler. | Honoured |
| Meta-ExternalFetcher | Meta | Fetches a link on demand for Meta AI. Documented by Meta as an on-demand fetcher. | Honoured |
| YouBot | You.com | Crawls to build the index You.com cites from. Documented by You.com as a search crawler. | Honoured |
Search index crawlers
Indexes your pages for a search engine whose results now feed AI answers.
| Bot | Operator | What it does | robots.txt |
|---|---|---|---|
| Googlebot | Indexes your pages for Google Search, which now feeds AI Overviews. | Honoured | |
| Bingbot | Microsoft | Indexes pages for Bing and Microsoft Copilot. | Honoured |
| Applebot | Apple | Indexes pages for Siri and Spotlight Suggestions. | Honoured |
| DuckDuckBot | DuckDuckGo | Indexes pages for DuckDuckGo search. | Honoured |
Training crawlers
Crawls public content to train future AI models. Blocking it is a policy choice, not a visibility fix.
| Bot | Operator | What it does | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Crawls public web content to train future OpenAI models. | Honoured |
| ClaudeBot | Anthropic | Crawls public content to train Anthropic’s Claude models. | Honoured |
| Google-Extended | Controls whether your content trains Gemini and Vertex AI. A robots.txt token, not a separate crawler. | Honoured | |
| Meta-ExternalAgent | Meta | Crawls content to train Meta’s Llama models. | Honoured |
| Applebot-Extended | Apple | Controls whether your content trains Apple Intelligence. A robots.txt token. | Honoured |
| Amazonbot | Amazon | Crawls content for Amazon’s AI (Alexa, Rufus) and search. | Honoured |
| Bytespider | ByteDance | Crawls content to train ByteDance / TikTok AI models. Historically aggressive; verify it honours robots.txt. | Verify |
| CCBot | Common Crawl | Builds the open Common Crawl dataset used to train many LLMs. | Honoured |
| Cohere-AI | Cohere | Crawls content for Cohere’s language models. | Honoured |
On-demand fetchers
Fetches specific pages on demand for its operator.
| Bot | Operator | What it does | robots.txt |
|---|---|---|---|
| Anthropic-AI | Anthropic | Legacy Anthropic fetcher user-agent. | Honoured |
| GoogleOther | Google’s general-purpose fetcher for internal R&D and one-off crawls. | Honoured |
Which of these can actually read your site?
The free crawler access checker reads your robots.txt and fetches a page as each retrieval crawler, so you can see the policy and the reality side by side.