Googlebot
Indexes your pages for Google Search, which now feeds AI Overviews.
| Operator | |
| Role | Search index crawlerIndexes your pages for a search engine whose results now feed AI answers. |
| User agent token | Googlebot |
| robots.txt | The operator documents honouring robots.txt. |
| Operator docs | developers.google.com |
Allow or block Googlebot in robots.txt
Add either rule to the robots.txt at your site root. robots.txt is a request, not a lock: this operator documents honouring it, so the rule is normally enough.
Allow
User-agent: Googlebot Allow: /
Block
User-agent: Googlebot Disallow: /
Other Google crawlers
- Google-ExtendedTraining crawler
- GoogleOtherOn-demand fetcher
Is Googlebot allowed on your site right now?
The free checker reads your robots.txt and probes your site as each AI crawler, so you see the policy and the reality in one pass.