Free tool
Can AI crawlers read your site?
Check what your robots.txt tells GPTBot, ClaudeBot, PerplexityBot and the rest, then watch what actually happens when each crawler requests a page. Free, no signup, takes seconds.
Not sure what a crawler in your logs does? The AI bot directory documents every major one: who runs it, what it feeds, and how to allow or block it.
Questions this tool gets asked
What does the checker actually do?
Three things. It reads your robots.txt and reports what it tells each major AI crawler. It probes for a sitemap and an llms.txt file. And it fetches a page from your site while presenting each retrieval crawler’s user agent, compared against a normal browser baseline, which catches firewall blocks robots.txt parsing can never see.
Which crawlers does it check?
The retrieval crawlers that decide whether AI can read you live, including OAI-SearchBot for ChatGPT, Claude-SearchBot, PerplexityBot, Google-Extended and Bingbot, plus the training crawlers GPTBot, ClaudeBot, CCBot and Amazonbot. The AI bot directory documents each one.
Should I block training crawlers?
That is a policy choice, not a visibility fix. Blocking training bots like GPTBot does not remove you from AI answers today, and blocking retrieval bots almost certainly hurts you. The checker separates the two groups so you can decide per group.
My firewall rejected your probe. Does that mean the real crawler is blocked?
Not necessarily. Our probe presents the crawler’s user agent but comes from our servers. A firewall that verifies crawler IP ranges may reject our probe while serving the real crawler. The result says so wherever that ambiguity exists.