AI Robots.txt Generator

Robots.txt access is binary: a named crawler either can read your site or it cannot, and each of the 16 bots listed above is matched by its own exact name, not by a single blanket rule. This tool writes one group per bot so the intent is explicit, then adds a wildcard group at the end for every other crawler, including ordinary search engines.

Every name in the list, and the one-line description next to it, was checked against that company's own current documentation before publishing, listed below. Two are worth calling out plainly: anthropic-ai has no current documentation of its own, since Anthropic's support article now documents only ClaudeBot, Claude-User, and Claude-SearchBot, and industry reporting describes anthropic-ai as retired in that bot's favor. Bytespider has no reachable official ByteDance documentation at all; its row says so rather than pointing at a source that does not exist.

A wildcard rule that disallows everything also disallows every AI crawler on this list by default, unless a more specific rule names that bot separately. That is the single most common accidental block, and it is why this tool always writes an explicit line for each named bot rather than leaving it to inherit from the wildcard group.