Overview
AI Crawler robots.txt Checker is a free, no-signup tool from AI Ranked, a publication on AI search visibility. Paste any robots.txt and the checker reports which of 21 AI user agents it allows or blocks: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, Amazonbot and more.
Most robots.txt files were written for Googlebot and Bingbot years ago. A blanket rule aimed at an old scraper can quietly keep your pages out of ChatGPT search, Perplexity answers and Google AI features today, and being blocked never shows up as an error. The checker makes it visible in seconds.
Key features
21 AI crawlers covered. Training crawlers, search indexers and on-demand user fetchers, because each type plays a different role in whether an AI product can read and cite your site.
The rule behind every result. For each crawler the tool shows the exact line in your file that produced the outcome, so you can see whether a bot is blocked by name or caught by a wildcard rule.
Ready-made rules to copy. Copy snippets to allow or block specific bots and paste them straight into your file.
Runs entirely in your browser. No account, no signup and no server round trip. Nothing you paste is sent to a server, so staging files and client sites stay private.
How to use it
- Open yoursite.com/robots.txt and copy the full text.
- Paste it into the checker.
- Read the allowed and blocked status for each of the 21 AI user agents, with the rule that decided it.
- Copy the suggested rules, update your file, and run the check again to confirm the change.
Who it is for
SEO and content teams who want their pages cited in AI answers, developers who maintain robots.txt across many sites, publishers deciding on an AI crawler policy, and agencies running technical audits.
Why it matters
The checker was built alongside AI Ranked's crawler access study of 261 major websites, which found that 22.6% of them block GPTBot. Many also block other AI agents through wildcard rules that were probably never intended to have that effect. The study's data is open, and the checker lets you run the same analysis on your own site.
About AI Ranked
AI Ranked covers how ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews choose which sources to cite. It publishes sourced guides, original research, a glossary, and free no-signup tools: a citability grader, this robots.txt checker, an llms.txt generator and a schema generator.
Pricing
Free. No paid tier, no account, no usage cap.
