The AI Crawler List, With Every Robots.txt Rule

Eight named crawlers decide whether an AI engine can read your website, and most robots.txt files block some of them by accident. This is the full list: each agent by its exact user agent name, what it actually does, and the rule that allows or blocks it on purpose rather than by mistake.

An AI crawler is an automated fetcher operated by an AI company to collect web pages for training, for search, or for a live user request, and the three purposes are controlled separately. The AI crawlers glossary entry carries the definition; this page is the reference list.

OpenAI: GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User

OpenAI documents four separate user agents, and they do different jobs (OpenAI, 2026). GPTBot crawls pages that may be used to train future models. OAI-SearchBot is the one that surfaces your pages inside ChatGPT's live search feature, so blocking it can remove you from ChatGPT answers even though it has nothing to do with training. OAI-AdsBot checks ad landing pages and is unrelated to either. ChatGPT-User fires only when a live user's question makes ChatGPT open your specific page mid-conversation; OpenAI notes it is "not used for crawling the web in an automatic fashion," so blocking it can interrupt an active user's request rather than routine indexing.

To allow all three main bots, add three separate blocks to your robots.txt:

• User-agent: GPTBot then Allow: /
• User-agent: OAI-SearchBot then Allow: /
• User-agent: ChatGPT-User then Allow: /

Perplexity: PerplexityBot and Perplexity-User

Perplexity documents two bots (Perplexity, 2026). PerplexityBot is "designed to surface and link websites in search results on Perplexity" and, per Perplexity's own documentation, is not used to train its models. Perplexity-User fires when a live user's question makes Perplexity fetch your page for an answer, and Perplexity's documentation notes this fetcher "generally ignores robots.txt rules" since a real person, not an automated crawl, triggered it.

To allow it: User-agent: PerplexityBot then Allow: /

Anthropic: ClaudeBot, Claude-User, Claude-SearchBot

Anthropic operates three crawlers with different jobs (Anthropic, 2026). ClaudeBot collects content that may contribute to model training. Claude-User fires when a person using Claude asks a question that makes it visit a specific page. Claude-SearchBot crawls to improve the relevance of Claude's search-grounded answers. Anthropic publishes a verifiable IP list for all three at claude.com/crawling/bots.json.

To allow it: User-agent: ClaudeBot then Allow: /

Google-Extended

Google-Extended is not a separate crawler. It is a token that controls what Google may do with content Googlebot already crawled. Disallowing it opts your content out of training and grounding for Gemini Apps and the Vertex AI API, without removing you from Google Search. Google states plainly that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google, 2026).

To opt out of AI training only: User-agent: Google-Extended then Disallow: /

CCBot

CCBot is the crawler behind Common Crawl, the nonprofit open web archive whose stated mission is an "open repository of web crawl data that is universally accessible" (Common Crawl, 2026). Common Crawl data has been a documented, major component of large language model training sets going back to GPT-3, so blocking CCBot is a reasonable default if you want to opt out of general-purpose training corpora, distinct from any single company's own crawler.

To block it: User-agent: CCBot then Disallow: /

Applebot-Extended

Applebot-Extended is the AI-training counterpart to the base Applebot that already powers Siri, Spotlight, and Safari search results. Apple's own documentation is explicit that the two are separate: "Webpages that disallow Applebot-Extended can still be included in search results," because Applebot-Extended "does not crawl" on its own, it only governs how Apple may use data the base Applebot already collected (Apple, 2026).

To opt out of AI training only: User-agent: Applebot-Extended then Disallow: /

The wildcard trap

The most common mistake is a rule meant for search engines that accidentally blocks every AI crawler too. A wildcard block, User-agent: * followed by Disallow: /, is not automatically limited to traditional search engines. Google's own robots.txt specification confirms that crawlers match the most specific applicable rule for their name; if no rule names a bot specifically, the wildcard rule is the one that applies to it (Google, 2026). If your wildcard block disallows your whole site, every AI crawler listed above is blocked by default unless you add an explicit, separately named Allow rule for it.

What Google says about all of this

Worth saying plainly: Google's own guidance states you "don't need to create new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode (Google, 2026). Robots.txt access is about whether an engine can read you at all, not a ranking trick, and it is table stakes, not a growth hack. Check your live file for free with our AI crawler checker, or start from scratch with our LocalBusiness schema generator once access is confirmed. If a block turns out to be your problem, the six causes of absence covers what comes next, and appearing across every engine covers the engines beyond ChatGPT. Get your free AI visibility score once your crawlers are unblocked.