Free SEO, AI and GEO toolRobots.txt generator
Choose a template, your platform and, for each AI platform, what you allow: search, which lets you be cited, and model training. The generator writes the robots.txt file, commented line by line, ready to copy or download, with the steps to follow for your platform.
How to use the generator
- Choose a template: getting cited everywhere suits most businesses.
- Enter your platform, then the folders that crawlers don't need to visit, such as the basket or the customer account.
- Set the AI crawlers, one by one if you wish.
- Copy the file and follow the steps for your platform.
Search or training: what you're choosing
Each AI platform uses several crawlers. Search crawlers find the pages that the AI cites in its answers: blocking them means giving up on being cited. Training crawlers collect pages for future models: blocking them doesn't remove you from answers; OpenAI states this for its own crawlers, and Anthropic separates its crawlers in the same way. The AI crawlers and robots.txt guide explains the role of each one.
Before you put the file online
- Keep a copy of your current file.
- Once it's live, open yourwebsite.co.uk/robots.txt to check that it shows.
- Test it with the robots.txt tester for AI.
- Check the robots.txt report in Google Search Console.
- Allow about 24 hours for OpenAI to take the change into account, according to its documentation.
Frequently asked questions: robots.txt, SEO, AI and GEO
What is robots.txt used for?
The robots.txt file, placed at the root of a website, tells search engine and AI crawlers which pages they may crawl. The major crawlers follow it, but it doesn't protect any content: it's public.
How do I use the robots.txt file in WordPress?
WordPress creates a virtual file as long as there isn't one on the server. To replace it with the one from the generator, paste it into Yoast SEO, under Tools, then File editor, or into Rank Math, in Advanced Mode, under General Settings, then Edit robots.txt.
How do I create a robots.txt file?
With this generator: choose a template, your platform and, for each AI platform, what you allow. Then copy the file and upload it to the root of your website, at yourwebsite.co.uk/robots.txt, following the steps for your platform. Test it afterwards with the robots.txt tester for AI.
Is the generated file right for my website?
It's a safe starting point. If your website already has specific rules, add them to the generated file rather than losing them, and test the result.
Should I block training crawlers?
It's your choice, with no effect on your presence in answers as long as search crawlers stay allowed. The “Get cited without being used for training” template makes this setting for you.
