Playbook · AI search and GEO

Updated

How should your robots.txt treat AI crawlers?

Treat AI crawlers in your robots.txt by job. Allow the bots that power AI search, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, if you want to be cited, and decide separately whether to block training bots like GPTBot, ClaudeBot and the Google-Extended token.

In one month on Vercel’s network, GPTBot made 569 million fetches and Googlebot 4.5 billion.
Source: Vercel, 17 Dec 2024

OpenAI

  • OAI-SearchBot: shows sites in ChatGPT search. Sites that block it do not appear in ChatGPT search answers.
  • GPTBot: crawls content that may be used to train OpenAI’s models. Blocking it says your content should not be used for training.
  • ChatGPT-User: fetches pages for actions a user starts. OpenAI says robots.txt rules may not apply to it, and points to OAI-SearchBot for search opt-outs.
  • OAI-AdsBot: only visits pages submitted as ads on ChatGPT.
  • Each setting is independent. Changes can take about 24 hours to apply, so wait a day before you debug a rule that looks ignored.

Anthropic

  • ClaudeBot: collects content that may be used for training. Blocking it excludes your future content from training data.
  • Claude-SearchBot: crawls to improve Claude’s search results. Blocking it may reduce your visibility there.
  • Claude-User: fetches pages when a Claude user asks a question that needs them. Blocking it stops those fetches.
  • Anthropic supports the non-standard Crawl-delay rule, and asks you to add rules on every subdomain you want covered. It links a list of its crawler IP addresses, but warns that blocking by IP may not work as an opt-out, because the bot can then no longer read your robots.txt.
  • Since 30 October 2026 Anthropic has also begun to recognize a general “no AI training” preference signal, based on an IETF draft. A robots.txt rule that names ClaudeBot takes priority over it.

The other side of a block: why Claude can’t read Reddit

Google

  • Google-Extended is a token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in the Gemini apps and Vertex AI.
  • Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
  • AI Overviews and AI Mode follow normal Search controls. To limit what they show from a page, Google points to nosnippet, data-nosnippet, max-snippet or noindex.
  • To take the whole site out of them, Search Console has a Search generative AI control. Every site is included by default, and Google says the control is not a ranking signal for the rest of Search.

Perplexity

  • PerplexityBot: surfaces and links sites in Perplexity search. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it.
  • Perplexity-User: fetches pages for a user’s question and generally ignores robots.txt.

An example robots.txt: in AI search, out of training

Each pair goes on two lines in the real file. Think about the last line before copying it: Google-Extended also covers grounding in Gemini apps, so blocking it is more than a training opt-out. This site allows every bot, because I want to be read and cited.

  • User-agent: OAI-SearchBot / Allow: /
  • User-agent: Claude-SearchBot / Allow: /
  • User-agent: PerplexityBot / Allow: /
  • User-agent: GPTBot / Disallow: /
  • User-agent: ClaudeBot / Disallow: /
  • User-agent: Google-Extended / Disallow: /

What robots.txt cannot do

Google says robots.txt is mainly for managing crawler traffic, and is not a way to keep a page out of Google. Use noindex for that. Respectable crawlers obey robots.txt, other crawlers might not, and the user-triggered fetchers of OpenAI and Perplexity may skip it by design. A robots.txt is a polite sign on the door, and it works on visitors who read signs. If something must stay private, put it behind a login.

Read next

Sources

Start

Tell me where the product is stuck.

In 30 minutes we look at where things stand, what you need from a CTO right now, and which option fits. Either way, you leave with a clear next step.

How the call goes

  1. 01You tell me about the product, the team and the next 6 months.
  2. 02I ask about the code, the stack and what keeps breaking.
  3. 03We pick the option that fits, or agree that none does.

09:00 to 20:00 CET

Founding rate: the first 3 clients get 20% off, in return for a named testimonial and a case study. On a monthly plan, the 20% applies to the first month.