What are AI crawlers?

Why it matters to a local business

One plain text file, yoursite.com/robots.txt, is the first gate every crawler on this page meets before it can read anything else on your site. A veterinary clinic that never touches this file usually has no problem, but one that installs a security plugin or copies a boilerplate wildcard rule meant to stop generic scrapers can accidentally block a dozen different AI crawlers at once, becoming invisible to several AI engines without anyone deciding that on purpose.

How robots.txt controls each crawler

Every crawler below is controlled through the Robots Exclusion Protocol, formally standardized by the Internet Engineering Task Force as RFC 9309, which "specifies and extends" the original method "for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers" (IETF, 2022). Crawlers determine which rules apply to them "by finding in the robots.txt file the group with the most specific user agent that matches the crawler's user agent" (Google, 2026), which means a wildcard rule, User-agent: * followed by Disallow: /, catches every crawler below unless you add a separate, explicitly named Allow rule for the ones you want to keep open. The standard is also explicit that these rules "are not a form of access authorization" (IETF, 2022): a well-behaved crawler chooses to respect a disallow rule, but the file itself cannot technically enforce it the way a password can.

The crawlers, one by one

GPTBot (OpenAI)

GPTBot "is used to make our generative AI foundation models more useful and safe" by crawling "content that may be used in training" (OpenAI, 2026). It is separate from OpenAI's search and live-user bots below, and OpenAI publishes verifiable IPs at openai.com/gptbot.json. To block only training: User-agent: GPTBot, Disallow: /.

OAI-SearchBot (OpenAI)

OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features" (OpenAI, 2026), and OpenAI states directly that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" (OpenAI, 2026), making this the crawler that most directly controls live ChatGPT citation eligibility. Verify at openai.com/searchbot.json.

ChatGPT-User (OpenAI)

This agent fires only "for certain user actions in ChatGPT and Custom GPTs" and OpenAI states it "is not used for crawling the web in an automatic fashion" (OpenAI, 2026), so it behaves like a real visitor's browser opening one page during an active conversation rather than an automated crawl.

PerplexityBot (Perplexity)

PerplexityBot is "designed to surface and link websites in search results on Perplexity" and Perplexity states "it is not used to crawl content for AI foundation models" (Perplexity, 2026). A separate agent, Perplexity-User, fires on live user requests and "generally ignores robots.txt rules" since a person triggered it (Perplexity, 2026). Verify at perplexity.com/perplexitybot.json.

ClaudeBot (Anthropic)

ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training" (Anthropic, 2026). Anthropic runs two further agents, Claude-User for live requests and Claude-SearchBot to "improve search result quality" (Anthropic, 2026), each controlled separately. Anthropic publishes verifiable IPs at claude.com/crawling/bots.json.

Google-Extended (Google)

Not a crawler itself, but a robots.txt token governing reuse of content Googlebot already crawled. Google states plainly that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google, 2026); it only controls training and grounding for Gemini apps and Vertex AI.

CCBot (Common Crawl)

CCBot feeds Common Crawl, "a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data" (Common Crawl, 2026) that many AI labs reuse for training rather than crawling the web themselves. Block with User-agent: CCBot, Disallow: /. Common Crawl warns of frequent spoofing; verify against index.commoncrawl.org/ccbot.json.

Applebot-Extended (Apple)

Apple's opt-out token for AI training, separate from the base Applebot that powers Siri, Spotlight, and Safari search. Apple is explicit that "Applebot-Extended does not crawl webpages" and "webpages that disallow Applebot-Extended can still be included in search results" (Apple, 2026), since it only governs reuse of already-crawled data.

Bingbot (Microsoft)

Bingbot indexes for Bing search and feeds the retrieval layer behind Microsoft Copilot's web-grounded answers, identifying itself with a "bingbot/2.0" marker (Microsoft, 2026). Bing's documentation recommends its Verify Bingbot tool or a reverse DNS check against search.msn.com over trusting the user agent string alone.

Amazonbot (Amazon)

Amazon states Amazonbot "is used to improve our products and services," which "may be used to train Amazon AI models" for Alexa, search, and shopping features (Amazon, 2026). It respects the Robots Exclusion Protocol and honors page-level noarchive, noindex, and none meta tags. Block with User-agent: Amazonbot, Disallow: /.

meta-externalagent (Meta)

Meta's crawler for "training foundation AI models or improving products by indexing content directly" (Meta, 2026), identifying itself as meta-externalagent/1.1. Meta notes crawlers "may cache the contents of robots.txt for up to 24 hours," so a new rule can take a day to take effect.

Bytespider (ByteDance)

Bytespider, operated by ByteDance to gather training data for its own large language models, is a documented exception to the rest of this list. ByteDance publishes no official crawler documentation of its own. The community-maintained ai.robots.txt registry, a technical dataset widely used by webmasters, lists its robots.txt respect as "No" and its function as "LLM training," including "ChatGPT competitors" (ai.robots.txt, 2026). Independent site operators have reported it continuing to request paths listed under Disallow, so a robots.txt rule alone may not fully stop it; blocking at the server or firewall level is the more reliable option if it is straining your site.

What to do about it

• Check every crawler on this page at once with the free AI Crawler Checker, rather than guessing which ones a wildcard rule already caught.
• Write a separate, explicitly named rule for each crawler you want to control; a rule for one does not extend to another.
• Submit an XML sitemap so any crawler you allow can discover your pages efficiently rather than relying on internal links alone.
• Verify suspicious traffic against each company's published IP list where one exists, since user agent strings alone are trivial to spoof.
• Decide deliberately per bot: training-only crawlers (GPTBot, CCBot, Google-Extended, Applebot-Extended, Amazonbot, Bytespider) carry a different tradeoff than search and citation crawlers (OAI-SearchBot, PerplexityBot, Bingbot, Claude-SearchBot) if you block them.