Free SEO, AI and GEO toolRobots.txt tester for AI
Enter your website's address: the tester reads its robots.txt file and shows, AI by AI, which crawlers can read your pages. First the ones that let you be cited in answers, then the ones used for training. You can see at a glance whether ChatGPT, Claude, Perplexity or Google can cite you.
How to use the tester
- Enter your website's address; add a specific page if you want to test it.
- Read the summary: one line per AI platform tells you whether it can cite you.
- Open the details to see the rule applied to each crawler, and fix it with the robots.txt generator if needed.
How to read the result
Search crawlers matter most: if they're blocked, the AI can't use your pages in its answers. Training crawlers are your choice. User-request crawlers read a page when a user asks them to; some ignore robots.txt, and the tester flags this.
The crawlers tested
The crawlers published by OpenAI, Anthropic, Perplexity, Google, Mistral AI, Apple and Common Crawl, plus Bingbot, Bing's crawler. The list follows their official documentation; it's updated when a publisher adds or renames a crawler.
Going further
The tester reads your robots.txt file. Also check the settings of your firewall, your host or Cloudflare, which apply outside this file. For a noindex directive, use the on-page SEO checker. And to find out whether AI platforms already cite you, start the free AI SEO audit.
Frequently asked questions: robots.txt testing, SEO, AI and GEO
How do I test my robots.txt file?
For AI crawlers, this tester answers crawler by crawler. For Google, Search Console also offers a robots.txt report, in its settings, which shows the version Google has read and its errors.
How do I know if ChatGPT can read my website?
Check that your robots.txt allows OAI-SearchBot, then that no firewall blocks it. In your analytics, visits tagged “utm_source=chatgpt.com” show that ChatGPT is already recommending some of your pages.
My robots.txt allows everything: why doesn't AI cite me?
Being able to read a page isn't enough. AI cites pages that answer questions clearly, and businesses that other websites talk about. The free AI SEO audit shows who it cites instead of you, and why.
How do I fix a crawler blocked by robots.txt?
Create a new file with the robots.txt generator, then put it online following the steps for your platform. Then test it again.
Does the test follow Google's rules?
Yes. The tester applies the robots.txt standard as Google reads it: the most specific rule wins and, when rules are equally specific, allow takes precedence over disallow.
