Robots.txt and AI crawlers: the SEO and GEO settings to know
The robots.txt file tells crawlers which pages of your website they can crawl. AI platforms use two kinds: search crawlers, which find the pages to cite, and training crawlers. To be cited, let the first kind through; blocking the second is your choice.
Contents
What is the robots.txt file?
It's a simple text file, placed at the root of your website: yourwebsite.co.uk/robots.txt. Open this address in your browser and you'll see yours. It contains groups of rules: each group starts with the name of a crawler (User-agent), followed by the parts of the website it can crawl (Allow) or not (Disallow).
User-agent: * Disallow: /basket/ Sitemap: https://yourwebsite.co.uk/sitemap.xml
This file tells all crawlers (*) not to crawl the basket, and tells them where to find the list of the website's pages.
Two points to remember:
- It's an instruction, not a lock. Major crawlers follow it, but the file is public and protects nothing. To keep content private, you need a login.
- It controls crawling, not display. A disallowed page can still be known to a search engine if other websites link to it. To keep a page out of Google, use the noindex tag, not robots.txt.
AI crawlers: which ones get you cited
Each AI platform publishes the list of its crawlers. Here's what their official documentation says.
| AI platform | Crawler | Role | If you block it |
|---|---|---|---|
| ChatGPT | OAI-SearchBot | Find the pages shown in ChatGPT search | Your pages are no longer used in answers; at best a link and a title |
| ChatGPT | GPTBot | Collect pages to train OpenAI's models | Your pages are excluded from training, with no effect on search |
| ChatGPT | ChatGPT-User | Read a page at a user's request | According to OpenAI, robots.txt may not apply |
| Claude | Claude-SearchBot | Index pages for search | Reduced visibility in Claude's answers |
| Claude | ClaudeBot | Collect pages for training | Your future pages are excluded from training |
| Claude | Claude-User | Read a page at a user's request | Reduced visibility for searches requested by users |
| Perplexity | PerplexityBot | Show and link to websites in its results; no training | Perplexity no longer indexes the text of your pages; it can still show the domain, the title and a brief summary |
| Perplexity | Perplexity-User | Read a page at a user's request | According to Perplexity, this crawler generally ignores robots.txt |
| Googlebot | Crawl pages for Google Search, including AI Overviews and AI Mode | Google can no longer read your pages or cite them in its answers | |
| Google-Extended | Not a separate crawler: a control that governs the use of your pages to train Gemini and for some other Google AI systems | No effect on Google Search or on rankings, according to Google |
Also keep Bingbot, Bing's crawler, allowed: ChatGPT partly relies on third-party search engines.
Other crawlers to know about
| Company | Crawler | Role |
|---|---|---|
| Mistral AI | MistralAI-Index | Index pages for its assistant's search |
| Mistral AI | MistralAI-Training | Collect pages to train its models |
| Mistral AI | MistralAI-User | Read a page at a user's request, following robots.txt |
| Apple | Applebot | Crawl the web for Siri, Spotlight and Safari; its data may also be used for training, unless refused via Applebot-Extended |
| Apple | Applebot-Extended | Not a separate crawler: a control that refuses the training of Apple's models, with no effect on its search results |
| Common Crawl | CCBot | Build a public archive of the web, reused by many AI projects |
Mistral AI, the French AI company, follows the same logic as its competitors: one crawler for search, one for training, one for user requests.
Robots.txt: SEO, AI and GEO settings for each choice
You want to be cited everywhere
This is the right choice for most businesses. A file that lets everything through, except pages of no interest, is enough:
User-agent: * Disallow: /basket/ Sitemap: https://yourwebsite.co.uk/sitemap.xml
You want to be cited, without being used for training
User-agent: * Disallow: /basket/ User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / Sitemap: https://yourwebsite.co.uk/sitemap.xml
A crawler follows the group that bears its name, otherwise the * group. Here, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot follow the first group and remain allowed. Worth knowing: blocking Google-Extended can also reduce the use of your pages in some of Gemini's answers.
You want to protect part of your website
Disallow only the folders concerned, for example /account/, and above all protect them with a login: robots.txt is public and hides nothing.
A robots.txt file, annotated
Here's a file of the kind often found after a copy-and-paste “to block AI”:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / Sitemap: https://yourwebsite.co.uk/sitemap_index.xml
- The first group is WordPress's: it closes the admin area and lets the rest through. Nothing to change.
- GPTBot: the training of OpenAI's models is refused. It's a choice, with no effect on search.
- OAI-SearchBot: this is the problem. ChatGPT can no longer use your pages in its answers. Delete these two lines.
- PerplexityBot: the same for Perplexity, whose search crawler it is. Delete these two lines too.
- Sitemap: the address of the sitemap, useful to all crawlers. Keep it.
The mistakes that make your website invisible to AI
- The forgotten “Disallow: /”. Added while the website was being built, to hide the test version, it blocks all crawlers, AI included, once the website is live.
- A block outside robots.txt. Some hosts, firewalls or services such as Cloudflare block AI crawlers in one click. At Cloudflare, since 15 September 2026, the “Block” and “Block on pages with ads” options also stop multi-purpose crawlers such as Googlebot, Bingbot or Applebot, search included; “Disallow AI Training” only refuses training. For a website that shows ads, Cloudflare suggests this block on ad pages at sign-up. Your robots.txt can be perfect and your pages still inaccessible.
- A Search Console setting. Since 31 August 2026, Google has let you exclude a website from its generative AI features, in the Search Console settings. When the setting excludes your website, it removes your pages from AI Overviews and AI Mode, even if they are properly indexed.
- A file in the wrong place. It must be at the root of each address: yourwebsite.co.uk/robots.txt, not in a subfolder.
- Blocked CSS or JavaScript files, which stop crawlers from displaying your pages the way your visitors see them.
- Using robots.txt to hide a page. It doesn't remove the page from Google: use the noindex tag instead.
How to check your robots.txt file
- Open yourwebsite.co.uk/robots.txt and look for “Disallow: /” lines, as well as the names of the crawlers in the table.
- In Google Search Console, the robots.txt report, in Settings, shows the version Google has read and any errors.
- If your website goes through Cloudflare or a firewall, check its AI crawler setting.
- Your host's logs show which crawlers actually visit your pages.
On WordPress, you edit the file from Yoast SEO or Rank Math: the AI SEO for WordPress page explains how. iasyo's free audit automatically checks each AI crawler's access to your website.
Frequently asked questions: robots.txt, SEO, AI and GEO
What is robots.txt?
The robots.txt file, placed at the root of a website, tells search engine and AI crawlers which pages they can crawl. It's a public instruction, followed by major crawlers, but it doesn't protect any content.
How do I use the robots.txt file in WordPress?
WordPress creates a virtual robots.txt file as long as there isn't one on the server. To edit it, use your SEO plugin: in Yoast SEO, Tools then File editor; in Rank Math, in Advanced Mode, General Settings then Edit robots.txt.
Should I block GPTBot?
It's your choice. GPTBot collects pages to train OpenAI's models. Blocking it doesn't remove you from ChatGPT search, as long as OAI-SearchBot remains allowed: OpenAI states that the two settings are independent.
Does blocking AI training remove me from AI answers?
No, if you let their search crawlers through: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity and Googlebot for Google. Google-Extended has no effect on Google Search.
How long does it take for a change to be taken into account?
About 24 hours for OpenAI, according to its documentation. Google also generally caches the file for up to 24 hours.
Does robots.txt protect my content from AI?
No. It's a public instruction, followed by major crawlers, not a protection. Crawlers that read a page at a user's request, such as Perplexity-User, may ignore it. To protect content, you need a login or a firewall.
Where is the robots.txt file?
At the root of your website: yourwebsite.co.uk/robots.txt. Each subdomain has its own: shop.yourwebsite.co.uk needs its own file.
What is ClaudeBot?
It's the crawler Anthropic uses to collect pages for training its models. Blocking it excludes your future pages from training. For Claude's answers, Anthropic relies on Claude-SearchBot and Claude-User: according to Anthropic, blocking those two can reduce your visibility in Claude.
About the author
Sources
Google, crawling documentation: “Introduction to robots.txt” and “How Google interprets the robots.txt specification”
Google Search Central, “Google's common crawlers” (Google-Extended)
OpenAI, “Overview of OpenAI Crawlers”
Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”
Perplexity, “Perplexity Crawlers”; Mistral AI, “Robots”; Apple, “About Applebot”; Common Crawl, FAQ
IETF, RFC 9309 “Robots Exclusion Protocol”, 2022
Cloudflare, “Your site, your rules: new AI traffic options for all customers”, 1 July 2026; Google, Search Console Help on the generative AI setting for Search
All accessed 28 September 2026
