Free tool · Answer engine optimization
AI Crawlability Checker
To find out whether AI assistants can read your site, check the rules your robots.txt gives each of their crawlers. Enter your domain below to see, bot by bot, whether OpenAI, Anthropic, Perplexity and Google bots may read your site root, which rule group applied, and what each vendor says the token controls.
Free, no account required.
The bots, in the vendors’ own words
Read from each vendor’s crawler documentation on 4 September 2026. The token is what goes on the User-agent line. The distinction that matters is search versus training: a training opt-out does not remove you from answers, a search block does.
| Token | Vendor | Kind | What it controls |
|---|---|---|---|
| OAI-SearchBot | OpenAI | search | Whether ChatGPT search can cite you. source |
| GPTBot | OpenAI | training | Training use only, per OpenAI; not search results. source |
| ChatGPT-User | OpenAI | user | Whether ChatGPT can read a page a user points it at. source |
| Claude-SearchBot | Anthropic | search | Whether Claude search can index and cite you. source |
| Claude-User | Anthropic | user | Whether Claude can read your pages during a user's session. source |
| ClaudeBot | Anthropic | training | Training use only, per Anthropic. source |
| PerplexityBot | Perplexity | search | Whether Perplexity can index you. source |
| Perplexity-User | Perplexity | user | Whether Perplexity can read a page during an answer. source |
| Google-Extended | training | Gemini training and grounding, per Google; not Search. source | |
| Googlebot | search | Google Search, and therefore AI Overviews and AI Mode. source |
Pair this with the llms.txt generator and, once the bots can read you, the AI visibility checker.
How to check whether AI bots can read your site
Step 1
Enter your domain
The root domain is enough. The checker fetches robots.txt over https, falls back to http, follows up to three redirects, and refuses local or internal addresses.
Step 2
Read the search bots first
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are the tokens that decide whether an engine can index and cite you. A block there is a visibility problem. A block on GPTBot, ClaudeBot or Google-Extended is a training opt-out, per each vendor, and does not remove you from answers.
Step 3
Check the rule group
A bot with no group of its own falls under the star group. If your star group disallows the site, every bot without an explicit allow is blocked, including the ones you meant to let in.
Step 4
Fix the file, then check what the engines say
Being readable is the floor, not the goal. Once the bots you want can read the site, run a real buyer question through the AI visibility checker to see whether the engines recommend you.
When you outgrow this tool
Being readable is the floor. Answer Radar tells you whether the engines then recommend you, and which pages they cite instead.
This tool is free and complete on its own, with no account and no limits worth mentioning. Linkeddit is what you use when the job stops being a one-off and starts needing to run every week without you.
FAQ
AI Crawlability Checker questions
What each token controls according to its vendor, and what a robots.txt check cannot tell you.
Does blocking GPTBot remove my site from ChatGPT answers?
Not according to OpenAI. Its crawler documentation says GPTBot governs training use and OAI-SearchBot governs ChatGPT search results, and that each setting is independent. A site can disallow GPTBot and allow OAI-SearchBot to stay in search answers while opting out of training. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.
What is the difference between ClaudeBot, Claude-User and Claude-SearchBot?
Anthropic documents three: ClaudeBot collects content that may contribute to training; Claude-User fetches pages when a Claude user asks a question; Claude-SearchBot indexes content to improve search results. Blocking the latter two reduces your visibility in Claude; blocking ClaudeBot alone is a training opt-out.
What does Google-Extended control?
Google describes it as a standalone control token, not a crawler: it manages whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. It has no effect on inclusion or ranking in Google Search, which Googlebot governs, and AI Overviews and AI Mode draw on the Search index.
Why does the checker only evaluate the site root?
Because the question a site owner is asking is whether a bot can read the site at all. The tool applies RFC 9309 longest-match rules to the root path and reports partial when a group allows the root but disallows some paths. Path-by-path testing is a different tool.
Can this tell me whether the bots actually visit, or whether I get cited?
No. robots.txt is a request that well-behaved crawlers honour; it does not show visits, and a firewall or CDN rule can block a bot the file allows. Whether an engine cites you is a separate measurement: the AI visibility checker tests one question, and Answer Radar tracks a prompt set across four engines.
Should I also have an llms.txt file?
It does not hurt and it is cheap, so the checker looks for one. The evidence that engines read it is thin, and we say so in our llms.txt guide. Allowing the search bots in robots.txt matters more than any llms.txt content.
More free tools
Every tool on the shelf is free and needs no account. See all free tools.
Check whether AI assistants recommend your product for a real buyer question, and see which competitors they name instead.
Generate the buyer prompts AI assistants actually get asked about your category, grouped by funnel stage.
Build a valid llms.txt file for your site, including the citation and safety blocks most generators leave out.
See the web searches an AI engine actually runs before it answers a buyer question, straight from the engine, not a template.