Bytespider
Crawls content to train ByteDance / TikTok AI models. Historically aggressive; verify it honours robots.txt.
| Operator | ByteDance |
| Role | Training crawlerCrawls public content to train future AI models. Blocking it is a policy choice, not a visibility fix. |
| User agent token | Bytespider |
| robots.txt | The operator’s robots.txt compliance has been questioned. Verify against your own logs. |
Allow or block Bytespider in robots.txt
Add either rule to the robots.txt at your site root. robots.txt is a request, not a lock: verify the effect in your logs, since this operator’s compliance has been questioned.
Allow
User-agent: Bytespider Allow: /
Block
User-agent: Bytespider Disallow: /
Is Bytespider allowed on your site right now?
The free checker reads your robots.txt and probes your site as each AI crawler, so you see the policy and the reality in one pass.