Short answer: treat AI crawlers by purpose. Search crawlers such as OAI-SearchBot and PerplexityBot help your pages appear in AI search answers; training controls such as GPTBot and Google-Extended govern whether your content is used to train models. Allow the ones you need for visibility, decide deliberately about training, and verify in your server logs that the rules work.
Why robots.txt matters more in AI search
Robots.txt has always told crawlers which parts of a site they may access. In AI search it has a new consequence: a blocked search crawler means your pages cannot be retrieved or cited in that platform’s answers. Many sites block AI crawlers without realising it, either through old blanket rules or through firewall and CDN “bot protection” settings.
The main AI user agents
| User agent / token | Operator | Purpose |
|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | |
| Google-Extended | Controls use of content for Gemini model improvement; does not affect Search | |
| OAI-SearchBot | OpenAI | Discovering pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetching pages on behalf of users |
| GPTBot | OpenAI | Collecting data for model training |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Bingbot | Microsoft | Bing search and Copilot |
| ClaudeBot | Anthropic | Collecting data for model training |
Names and behaviours change, so always check each provider’s current documentation. An expanded list is in AI Crawler List: User Agents Every Site Owner Should Know.
Example policies
Allow AI search, opt out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Allow everything
User-agent: *
Allow: /
Disallow: /wp-admin/There is no universally correct policy. Publishers with valuable proprietary content may opt out of training; most businesses that want visibility should at least allow search crawlers.
Robots.txt is not the whole story
- Firewalls and CDNs may block unfamiliar bots before robots.txt is ever read. Check bot-management settings.
- Rendering – if content only appears after JavaScript runs, some crawlers may not see it. See JavaScript SEO for AI Crawlers.
- Snippet controls – Google applies
nosnippetandmax-snippetto AI features as well as normal snippets. - Robots.txt is voluntary – reputable crawlers follow it; it is not an access-control system.
How to verify your setup
- Fetch
/robots.txtand read it line by line. - Search your server or CDN logs for each user agent and check the response codes.
- Test a key page with Google’s URL Inspection tool.
- Ask the relevant assistant about content that only exists on your site and see whether it can find it.
A decision framework
Before editing robots.txt, answer three questions for your business:
- Do we want to appear in AI search answers? For most businesses the answer is yes – so allow the search crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot).
- Are we comfortable with our content being used to train models? This is a business and values decision. Publishers with proprietary content often say no; many service businesses see little downside in allowing it.
- Are there sections that should never be crawled? Admin areas, internal search results, staging copies and customer portals should be disallowed for every crawler.
Write down the decision and the reason. When a new crawler appears – and new ones appear regularly – the policy makes the next decision quick.
What each choice affects
| If you block… | You are likely to lose… | You keep… |
|---|---|---|
| Googlebot | Google Search and AI Overviews entirely | Nothing in Google |
| Google-Extended | Use of your content for Gemini model improvement | Google Search, AI Overviews and AI Mode |
| OAI-SearchBot | Discovery in ChatGPT search | Other platforms |
| GPTBot | Use of your content for OpenAI model training | ChatGPT search (via OAI-SearchBot) |
| PerplexityBot | Citations in Perplexity | Other platforms |
Managing robots.txt on WordPress
WordPress generates a virtual robots.txt if no physical file exists. SEO plugins such as Yoast and Rank Math let you edit it from the dashboard; the Artificial AI SEO theme also appends explicit allow rules for AI search crawlers. If a physical robots.txt file exists in your site root, it overrides the virtual one – a common reason edits “don’t work”. After any change, open yoursite.com/robots.txt in a private window to confirm the live version.
Firewalls and CDNs: the hidden blocker
Services such as Cloudflare, Sucuri and hosting-level firewalls often challenge or block automated traffic by default, including legitimate AI search crawlers. Robots.txt says “you may enter”; the firewall slams the door anyway. Review bot-management settings, allow verified crawlers and check that responses to them are 200, not 403 or a challenge page.
Monitoring over time
Set a quarterly reminder to re-check robots.txt, firewall rules and logs. Site migrations, new security plugins and CDN changes frequently undo a careful policy without anyone noticing. A monthly log report showing hits from each AI crawler is an easy early warning.
Verifying that a crawler is genuine
Anyone can put “OAI-SearchBot” or “Googlebot” in a user-agent string. Scrapers often impersonate well-known crawlers to get past filters. Major operators publish ways to verify their bots: Google and Bing support reverse DNS lookups that resolve to their own domains, and OpenAI and Perplexity publish the IP ranges their crawlers use. Good firewalls can allow verified crawlers automatically while still blocking impostors. If you review logs manually, check a sample of requests against the published ranges before trusting the numbers.
Staging sites and duplicate environments
Staging and development copies should block every crawler – and should also be protected by a password, because robots.txt alone does not keep pages out of every system. The most common disaster is the reverse: a staging robots.txt with Disallow: / copied to the live site during a launch. Add a launch checklist item to open the live robots.txt and confirm it matches your policy.
A real-world scenario
A B2B software company noticed that it had disappeared from Perplexity answers it used to appear in, while competitors were still cited. Its robots.txt looked fine. The cause was a new firewall rule that challenged “unverified bots” – including PerplexityBot and OAI-SearchBot – with a JavaScript check the crawlers could not pass. The logs showed hundreds of 403 responses to both. After the team allowed verified AI search crawlers in the firewall, the crawlers returned within days, and citations gradually recovered over the following weeks. The lesson: always check logs, not just the robots.txt file.
Common mistakes
- A leftover
Disallow: /from a staging site. - Blocking OAI-SearchBot while intending to block only GPTBot.
- Assuming Google-Extended affects Google Search – it does not.
- Returning 403 errors to bots through aggressive firewall rules.
What about llms.txt?
llms.txt is a proposed file that gives AI systems a summary of your site. It is not a standard and does not replace robots.txt, but it is harmless to publish. See What Is llms.txt and Should You Use It?.
Frequently asked questions
Will blocking GPTBot hurt my visibility in ChatGPT search?
OpenAI documents OAI-SearchBot as the crawler for ChatGPT search, separate from GPTBot. Blocking GPTBot is intended to opt out of training, not search.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are part of Google Search, which uses Googlebot.
Get help
Our Technical AI SEO service audits crawler access, firewall rules and rendering, and implements a deliberate policy. Start with an AI SEO Audit.
