How should your robots.txt treat AI crawlers?
Treat AI crawlers in your robots.txt by job. Allow the bots that power AI search, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, if you want to be cited, and decide separately whether to block training bots like GPTBot, ClaudeBot and the Google-Extended token.
In one month on Vercel’s network, GPTBot made 569 million fetches and Googlebot 4.5 billion.
OpenAI
- OAI-SearchBot: shows sites in ChatGPT search. Sites that block it do not appear in ChatGPT search answers.
- GPTBot: crawls content that may be used to train OpenAI’s models. Blocking it says your content should not be used for training.
- ChatGPT-User: fetches pages for actions a user starts. OpenAI says robots.txt rules may not apply to it, and points to OAI-SearchBot for search opt-outs.
- OAI-AdsBot: only visits pages submitted as ads on ChatGPT.
- Each setting is independent. Changes can take about 24 hours to apply, so wait a day before you debug a rule that looks ignored.
Anthropic
- ClaudeBot: collects content that may be used for training. Blocking it excludes your future content from training data.
- Claude-SearchBot: crawls to improve Claude’s search results. Blocking it may reduce your visibility there.
- Claude-User: fetches pages when a Claude user asks a question that needs them. Blocking it stops those fetches.
- Anthropic supports the non-standard Crawl-delay rule, and asks you to add rules on every subdomain you want covered. It links a list of its crawler IP addresses, but warns that blocking by IP may not work as an opt-out, because the bot can then no longer read your robots.txt.
- Since 30 October 2026 Anthropic has also begun to recognize a general “no AI training” preference signal, based on an IETF draft. A robots.txt rule that names ClaudeBot takes priority over it.
- Google-Extended is a token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in the Gemini apps and Vertex AI.
- Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
- AI Overviews and AI Mode follow normal Search controls. To limit what they show from a page, Google points to nosnippet, data-nosnippet, max-snippet or noindex.
- To take the whole site out of them, Search Console has a Search generative AI control. Every site is included by default, and Google says the control is not a ranking signal for the rest of Search.
Perplexity
- PerplexityBot: surfaces and links sites in Perplexity search. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it.
- Perplexity-User: fetches pages for a user’s question and generally ignores robots.txt.
An example robots.txt: in AI search, out of training
Each pair goes on two lines in the real file. Think about the last line before copying it: Google-Extended also covers grounding in Gemini apps, so blocking it is more than a training opt-out. This site allows every bot, because I want to be read and cited.
- User-agent: OAI-SearchBot / Allow: /
- User-agent: Claude-SearchBot / Allow: /
- User-agent: PerplexityBot / Allow: /
- User-agent: GPTBot / Disallow: /
- User-agent: ClaudeBot / Disallow: /
- User-agent: Google-Extended / Disallow: /
What robots.txt cannot do
Google says robots.txt is mainly for managing crawler traffic, and is not a way to keep a page out of Google. Use noindex for that. Respectable crawlers obey robots.txt, other crawlers might not, and the user-triggered fetchers of OpenAI and Perplexity may skip it by design. A robots.txt is a polite sign on the door, and it works on visitors who read signs. If something must stay private, put it behind a login.
Read next
Sources
- OpenAI, overview of OpenAI crawlers, seen 17 Nov 2026
- Anthropic, how site owners can block the crawler, updated 4 Nov 2026
- Google Search Central, Google’s common crawlers, updated 30 Oct 2026
- Google Search Central, AI features and your website, updated 10 Dec 2025
- Search Console Help, Search generative AI control, seen 17 Nov 2026
- Google Search Central, introduction to robots.txt, updated 10 Dec 2025
- Perplexity, Perplexity crawlers, seen 17 Nov 2026
- Vercel, the rise of the AI crawler, 17 Dec 2024