Guides · Updated 3 November 2026
robots.txt: allow AI search bots, block training bots
You can let AI assistants find your products without handing over your content for training AI models. Most AI companies use separate bots for search and for training, and you control them separately in your robots.txt.
Which bot does what
| Bot | From | Used for | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search in ChatGPT | You do not appear in ChatGPT search answers |
GPTBot | OpenAI | Training AI models | Your content is not used for training |
ChatGPT-User | OpenAI | Visits a page when a user asks it to | robots.txt may not apply |
Claude-SearchBot | Anthropic | Search index for Claude | Your pages are not indexed for search |
Claude-User | Anthropic | Fetches pages for user questions | Claude cannot fetch your page |
ClaudeBot | Anthropic | Training AI models | Your future content is not used for training |
PerplexityBot | Perplexity | Perplexity search results (not training) | You do not appear in Perplexity |
Perplexity-User | Perplexity | Visits a page when a user asks it to | Generally ignores robots.txt |
Google-Extended | Gemini training and the use of your pages in Gemini answers | No effect on Google Search, but it does affect Gemini |
An example for online stores
This robots.txt allows the search bots, blocks the training bots of OpenAI and Anthropic, and keeps all bots out of your cart and checkout:
# AI search bots may read the shop, training bots may not
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Everyone else, including OAI-SearchBot, Claude-SearchBot and PerplexityBot
User-agent: *
Disallow: /cart
Disallow: /checkout
Sitemap: https://www.example-shop.com/sitemap.xmlWhat to watch out for
- Never block everything.
User-agent: *withDisallow: /also keeps out the search bots, and with them ChatGPT, Claude and Perplexity. - Google-Extended is a trade-off. Blocking it keeps your content out of Gemini training, but Gemini then cannot use your pages when answering questions either.
- Changes take time. OpenAI mentions about 24 hours.
- How to edit robots.txt depends on your store system. It is often done through the theme or an SEO plugin; ask your web developer if unsure.
The free Shelv scan shows straight away whether AI search bots can reach your product pages.
How does your product page score?
Paste the address of a product page into the free scan. Within seconds you see what AI assistants can and cannot read.