AI search engines like ChatGPT and Perplexity rely on dedicated crawlers to read your website before they can recommend you in their answers. If your robots.txt blocks these user agents, your content is effectively invisible to AI search results. To allow AI crawlers, you add specific `User-agent` and `Allow` directives for each known AI bot—covering 16 key user agents is a solid baseline for modern AI visibility.
---
Why robots.txt matters for AI search visibility
Robots.txt is a plain-text file at the root of your domain that tells crawlers which URLs they may or may not access. For human search engines, this file has been a standard for decades. For AI search engines, the same protocol applies—but many site owners have never updated their robots.txt to include AI-specific crawlers.
When an AI crawler is blocked, the AI search engine cannot index or analyze your content. That means your pages won't be cited, summarized, or recommended—even if you rank well in traditional Google results. Allowing AI crawlers is the first technical step in Generative Engine Optimization (GEO).
---
The 16 AI crawlers you should allow
Below is the commonly referenced list of AI user agents that you can permit in robots.txt. Note that exact crawler names may evolve over time—always check the official documentation for each provider before deploying.
| # | User-agent | Operated by |
|---|-----------|-------------|
| 1 | `GPTBot` | OpenAI (ChatGPT) |
| 2 | `ChatGPT-User` | OpenAI (ChatGPT interactive) |
| 3 | `OAI-SearchBot` | OpenAI (search) |
| 4 | `PerplexityBot` | Perplexity AI |
| 5 | `ClaudeBot` | Anthropic (Claude) |
| 6 | `anthropic-ai` | Anthropic (older agent) |
| 7 | `Google-Extended` | Google (AI training/answers) |
| 8 | `Google-CloudVertexBot` | Google Vertex AI |
| 9 | `CCBot` | Common Crawl (used by various AI) |
| 10 | `cohere-ai` | Cohere |
| 11 | `meta-externalagent` | Meta AI |
| 12 | `Applebot-Extended` | Apple (AI/Siri) |
| 13 | `Amazonbot` | Amazon (Alexa/AI) |
| 14 | `Bytespider` | ByteDance (TikTok AI) |
| 15 | `YouBot` | You.com |
| 16 | `DataForSeoBot` | DataForSEO |
---
Step-by-step robots.txt configuration
1. Locate your current robots.txt
Open `https://yourdomain.com/robots.txt` in a browser. If it does not exist, you'll see a 404. If it exists, copy its full content to a safe location.
2. Decide on allow vs. block
This guide assumes you want full accessibility for AI crawlers across your public pages. If you need to block specific paths (e.g., `/admin/` or `/checkout/`), you can combine `Allow` and `Disallow` rules per user agent.
3. Write the directives
Place the following block at the beginning of your robots.txt, before any general `User-agent: *` rules. This ensures AI crawlers are not affected by broader disallow rules.
```txt
AI search & crawler access
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: meta-externalagent
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Bytespider
Allow: /
User-agent: YouBot
Allow: /
User-agent: DataForSeoBot
Allow: /
```
4. Keep your existing rules intact
If you already have a `User-agent: *` block with `Disallow` rules, do not remove it—just ensure the above AI-specific blocks come first. Crawlers match the most specific user-agent rule, so your AI blocks will take precedence.
5. Test your file
After saving your changes, verify the file is accessible at `https://yourdomain.com/robots.txt`. You can also use a free robots.txt tester tool to check for syntax errors.
---
Common pitfalls when configuring AI crawlers
- Case sensitivity: User-agent names are case-sensitive. `gptbot` will not match `GPTBot`.
- Overly broad blocking: If your `User-agent: *` rule has `Disallow: /`, AI crawlers will be blocked unless you explicitly allow them—which you've done above. Just double-check the order.
- Forgetting new crawlers: AI crawler lists evolve. Review your robots.txt quarterly and add new user agents as they emerge.
- Blocking via meta tags: Some AI crawlers honor `<meta name="robots" content="noai">` tags. If you have those tags in your HTML, robots.txt alone will not make you visible.
---
DIY vs. hiring a service
If you are comfortable editing a text file and understand robots.txt syntax, the 15-minute DIY approach above is perfectly fine for a simple site. However, AI visibility goes far beyond robots.txt—you also need schema markup, content structure, and ongoing optimization for AI answers.
If you want a thorough evaluation of your AI search readiness, a professional check can catch issues beyond crawler access. Our AI Search Health Check is free and includes a full audit of your current AI visibility, including robots.txt configuration, content gaps, and technical signals. It's a no-cost way to know exactly where you stand before you invest in deeper changes.
---
Frequently Asked Questions
Can I block AI crawlers from specific pages while allowing them on the rest?
Yes. In each user-agent block, use `Allow: /public/` and `Disallow: /private/` to control access at the directory or URL level. Just remember that `Allow` must come before `Disallow` for the same path to take effect.
Does allowing AI crawlers hurt my Google ranking?
No. AI crawlers operate separately from Google's main indexing crawler. Allowing GPTBot or PerplexityBot does not change how Google ranks your pages. Many sites allow both without any ranking impact.
How often should I update my robots.txt for AI crawlers?
Check quarterly. AI companies announce new user agents or update existing ones frequently. A good practice is to subscribe to the official developer or SEO blogs of OpenAI, Anthropic, Perplexity, and Google to stay current.