Free tools / AI Crawler Checker
AI Crawler Checker
Enter your website and this tool reads your live robots.txt file and reports whether it blocks any of the eleven crawlers that feed ChatGPT, Claude, Perplexity, Gemini, Bing, and Apple Intelligence, with a ready-to-paste fix for anything blocked.
The AI Crawler Checker is a free AnswerFix tool that reads your live robots.txt file and reports whether it blocks any of the eleven crawlers that feed the major AI answer engines, including GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot and Claude-Web for Claude, PerplexityBot for Perplexity, Google-Extended for Gemini and AI Overviews, and Applebot-Extended for Apple Intelligence.
An AI engine can only cite facts it is allowed to read directly from your site. If your robots.txt disallows one of these bots, whether on purpose from an old security rule or by accident from a site builder's default settings, that engine is working from stale search-index data or nothing at all when it answers a question about your business. This is one of the few AI-visibility problems that is entirely binary: a bot is either allowed or blocked, there is no partial credit, and it is usually fixed with two lines of text.
Enter your website above and this tool fetches your robots.txt directly from your server, the same way these bots would, and checks the rules against each one using the same longest-match, wildcard-aware logic that real crawlers use to decide whether a page is off-limits. If a bot is blocked, you get a ready-to-paste Allow rule for it; if your site has no robots.txt at all, every bot is allowed by default and there is nothing to fix.
This check only reads a public text file at a well-known URL; it does not log in, does not crawl your whole site, and does not store the pages it reads. Checks are rate-limited per connection to keep the tool available and free for everyone.
Crawler checker questions
- Why would my robots.txt block AI crawlers in the first place?
- Most robots.txt files were written years before AI crawlers existed, so a blanket rule meant to stop scrapers or bad bots can accidentally catch GPTBot, ClaudeBot, or PerplexityBot too. Some site builders and security plugins also add broad bot blocks by default without listing which specific crawlers get caught.
- If a bot is blocked, does that mean I am invisible to that AI engine?
- It means that specific crawler cannot fetch your pages, which is a strong signal but not the whole picture; some engines also pull from search indexes or licensed data partners. Still, a blocked crawler is one less way that engine can verify your facts directly, so allowing it removes a real barrier.
- Is it safe to allow every bot on this list?
- Yes, for a public business website. Every bot listed here belongs to a known search or AI company and only reads publicly available pages, the same pages any visitor or search engine already sees. Allowing them costs nothing in server load for a typical small business site.
- How do I apply the fix this tool suggests?
- Copy the Allow rules shown after your check and paste them into your robots.txt file at your site root, replacing any Disallow line that currently names the same bot. Save the file, then run this checker again to confirm the bot now shows as allowed.
Now see whether ChatGPT recommends you
Being crawlable is step one. Get your free AI visibility score to see if you are actually being recommended.