All tools / Crawling & Access

AI Crawler Access Checker

Paste your robots.txt or point the tool at your site. It tests 17 AI crawlers against your rules and tells you exactly what each block means for your AI search visibility.

1Enter your site or paste robots.txtWe fetch yourdomain.com/robots.txt for you.

2We test every AI botEach user agent is evaluated against your rules the same way the crawler would.

3Get verdicts that mean somethingBlocked training bots are a choice. Blocked search bots cost you visibility.

Your site

Path to test (optional, defaults to the whole site)

AI bot	Purpose	Verdict	Why
GPTBot OpenAI	Model training
OAI-SearchBot OpenAI	AI search index
ChatGPT-User OpenAI	On-demand fetch
ClaudeBot Anthropic	Model training
Claude-SearchBot Anthropic	AI search index
Claude-User Anthropic	On-demand fetch
PerplexityBot Perplexity	AI search index
Perplexity-User Perplexity	On-demand fetch
Google-Extended Google	Model training
Bingbot Microsoft	AI search index
CCBot Common Crawl	Model training
Applebot-Extended Apple	Model training
Meta-ExternalAgent Meta	Model training
Amazonbot Amazon	AI search index
Bytespider ByteDance	Model training
DuckAssistBot DuckDuckGo	AI search index
MistralAI-User Mistral	On-demand fetch

This checks robots.txt rules only. CDN firewall rules, meta robots tags and X-Robots-Tag headers can also block bots and are not visible from robots.txt.

AI Crawler Access Checker

Related reading

Robots.txt ambiguities for AI crawlers: blocked or not?