AI crawler robots.txt generator

AI assistants can only cite pages their crawlers are allowed to read. This generates a robots.txt that explicitly welcomes the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more). Free, no sign-up.

Your robots.txt

Save this as robots.txt at your site root. It welcomes the major AI crawlers explicitly — a named user-agent group overrides the * group, so the rules are repeated for them.

# robots.txt — welcomes AI assistant & answer-engine crawlers
User-agent: *
Allow: /

# AI crawlers (a named group overrides '*', so the rules are repeated)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
Allow: /

Now see whether AI actually names your brand — free.

Check my brand — free

Frequently asked questions

Which AI crawlers should I allow?

The major ones are OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic's ClaudeBot and anthropic-ai; PerplexityBot; Google-Extended; and CCBot. Allowing them lets AI search index and cite your content.

Does robots.txt guarantee AI will cite my site?

No — it only permits crawling. Citation still depends on your content's authority, clarity, and relevance. Allowing the crawlers is the necessary first step.