Your server logs now contain a growing list of AI-related bots. Some collect content to train models, some index pages for AI search, and some fetch pages when a user asks a question. Treating them all the same is a mistake: blocking the wrong one can remove you from AI search results, while allowing everything may not suit your content strategy.
This guide lists the most important AI crawlers and user agents, explains what each does, and shows how to manage them. AI companies update their crawlers regularly, so always confirm details against each provider’s official documentation.
Three Types of AI Crawlers
- Training crawlers: collect public content that may be used to train AI models
- Search and indexing crawlers: build indexes used to surface and cite pages in AI search results
- User-triggered fetchers: retrieve a page because a user asked the AI to read or check it
Understanding the type helps you decide whether to allow or block each one.
The AI Crawler List
OpenAI
| User agent | Type | Purpose |
|---|---|---|
| GPTBot | Training | Collects content that may be used to train OpenAI models |
| OAI-SearchBot | Search | Indexes pages for ChatGPT search results and citations |
| ChatGPT-User | User-triggered | Fetches pages when users or actions in ChatGPT request them |
Anthropic
| User agent | Type | Purpose |
|---|---|---|
| ClaudeBot | Training | Collects content that may be used for model training |
| Claude-SearchBot | Search | Indexes content to improve search results for Claude |
| Claude-User | User-triggered | Fetches pages when Claude users request them |
Perplexity
| User agent | Type | Purpose |
|---|---|---|
| PerplexityBot | Search | Indexes pages for Perplexity answers and citations |
| Perplexity-User | User-triggered | Fetches pages for specific user requests |
| User agent or token | Type | Purpose |
|---|---|---|
| Googlebot | Search | Crawls for Google Search, including AI Overviews and AI Mode |
| Google-Extended | Control token | Controls use of content for Gemini training and certain grounding uses |
| GoogleOther | Other | Used by some Google product teams for non-Search crawling |
Microsoft
| User agent | Type | Purpose |
|---|---|---|
| Bingbot | Search | Crawls for Bing, which also supports Microsoft Copilot experiences |
Apple
| User agent or token | Type | Purpose |
|---|---|---|
| Applebot | Search | Crawls for Apple services such as Siri and Spotlight suggestions |
| Applebot-Extended | Control token | Controls use of content for Apple’s AI model training |
Others Worth Knowing
| User agent | Operator | Purpose |
|---|---|---|
| CCBot | Common Crawl | Builds open web datasets widely used in AI research and training |
| Meta-ExternalAgent | Meta | Crawls content for AI-related uses, including training |
| Amazonbot | Amazon | Crawls for Amazon services, including Alexa-related answers |
| Bytespider | ByteDance | Crawls content for ByteDance products and AI development |
| DuckAssistBot | DuckDuckGo | Fetches content for DuckDuckGo’s AI-assisted answers |
How to Decide What to Allow
If Visibility Is Your Priority
Allow search crawlers such as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, and Claude-SearchBot. These help your content appear and be cited in AI search. For more on why, see AI search visibility.
If Protecting Content Is Your Priority
Consider blocking training crawlers and control tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot, while still allowing search crawlers.
A Common Balanced Setup
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /Adjust based on your goals. Many marketing-focused sites allow training crawlers too, because broader presence can support brand awareness.
Important Caveats
- User-triggered fetchers may not follow robots.txt in the same way as automated crawlers. Some providers say so explicitly.
- User agents can be spoofed. Verify bots using the IP ranges or reverse DNS methods providers publish.
- Blocking is not retroactive. It does not remove content already collected.
- Firewalls matter. CDN and security tools can block bots even when robots.txt allows them.
How to Check Which Bots Visit Your Site
- Review server access logs for the user agents above
- Check response codes: 200 means success, 403 or 429 suggests blocking or throttling
- Verify IP addresses against official lists
- Review CDN bot analytics dashboards
- Confirm your live robots.txt reflects your intended policy
An AI SEO audit can include a full crawler access review.
Managing Crawlers on WordPress
On WordPress, robots.txt is often generated virtually by WordPress itself or by an SEO plugin such as Yoast SEO or Rank Math. Edit it through the plugin’s file editor, or upload a physical robots.txt file to your site root, which usually overrides the virtual version. Security plugins and hosting firewalls may add their own bot rules, so check those settings too. After any change, open yourdomain.com/robots.txt in a private browser window to confirm the live file shows exactly what you intended.
Keeping Your List Up to Date
New AI crawlers appear regularly, and existing ones change names or purposes. Review official documentation from major AI providers at least twice a year, and update your robots.txt and firewall rules accordingly. Document your crawler policy so your team knows why each decision was made.
Final Thoughts
AI crawlers are not all the same. Distinguishing between training, search, and user-triggered bots lets you protect what matters while staying visible in AI search. Build a deliberate policy, verify it in your logs, and review it regularly. For the wider technical picture, see our AI SEO checklist and guide to how AI search works.
Frequently Asked Questions
What are AI crawlers?
AI crawlers are bots operated by AI companies and search engines that collect content for model training, index pages for AI search results, or fetch pages when users ask AI assistants.
Which AI crawlers should I allow for visibility?
For AI search visibility, allow search crawlers such as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, and Claude-SearchBot. Blocking these reduces your chances of being surfaced and cited in AI answers.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may train OpenAI models, while OAI-SearchBot indexes pages for ChatGPT search results. You can block GPTBot and still allow OAI-SearchBot for search visibility.
Do all AI bots follow robots.txt?
Automated crawlers from major providers generally follow robots.txt, but some user-triggered fetchers may not. Use firewall or bot management controls if you need stronger restrictions on access.
How can I verify an AI crawler is genuine?
Compare IP addresses in your logs with the official IP ranges or reverse DNS methods published by each provider. Requests using a known user agent from other addresses may be spoofed.
How often should I update my AI crawler rules?
Review official crawler documentation and your robots.txt at least twice a year. New AI crawlers appear regularly, and providers sometimes change user agent names or purposes over time.
