Every AI bot that decides
whether AI can see your business.
Each AI company runs more than one agent, and they do different jobs: one builds a search index, one fetches a page when a user asks, one collects training data. Blocking the wrong one makes you invisible; blocking the right one is a legitimate choice. This guide lists each documented agent, what it does, whether it obeys robots.txt, and what to allow if you want to be found.
The agents, one by one
Descriptions are taken from each company's own documentation, linked in the sources at the end. Where a company says its user-triggered fetcher may ignore robots.txt, we quote that rather than soften it.
| Company | Agent | What it does | Obeys robots.txt? | If you block it |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Builds the index that surfaces websites in ChatGPT's search features | Yes | You disappear from ChatGPT search results |
| OpenAI | ChatGPT-User | Fetches a page when a user's request in ChatGPT or a custom GPT needs it. Not an automatic crawler | OpenAI says robots.txt rules "may not apply" because a user initiated the action | Live lookups of your pages can fail |
| OpenAI | GPTBot | Collects content that may be used to train OpenAI's models | Yes | Your content is excluded from future training; no effect on ChatGPT search |
| OpenAI | OAI-AdsBot | Validates the safety of pages submitted as ads on ChatGPT | Yes | Only relevant if you advertise on ChatGPT |
Googlebot | Crawls for Google Search, which also feeds AI Overviews and AI Mode | Yes | You leave Google Search entirely, AI features included | |
Google-Extended | A control token, not a separate crawler: governs whether content Googlebot already fetched may be used for Gemini training and grounding | Yes (as a robots.txt token) | Google states it does not affect inclusion or ranking in Search, and AI Overviews are not controlled by it | |
| Anthropic | Claude-SearchBot | Indexes content to improve Claude's search results | Yes | Reduced visibility in Claude-powered search answers |
| Anthropic | Claude-User | Fetches a page when a Claude user's question needs it | Yes | Claude cannot retrieve your pages for a user |
| Anthropic | ClaudeBot | Collects content that may contribute to model training | Yes | Excluded from training; the other two agents are unaffected |
| Perplexity | PerplexityBot | Surfaces and links websites in Perplexity search results; not used to train models | Yes | You disappear from Perplexity results |
| Perplexity | Perplexity-User | Visits a page to answer a user's question and may link to it | Perplexity states it "generally ignores" robots.txt because a user asked | Limited; it will usually still fetch |
| Microsoft | bingbot | Crawls for Bing, whose index grounds Microsoft Copilot answers | Yes | You leave Bing and Copilot. Use the nocache or noarchive meta tags to stay in Bing but limit use in AI answers |
| Apple | Applebot / Applebot-Extended | Applebot crawls for Siri and Spotlight; Applebot-Extended is the token that opts your content out of training Apple's foundation models | Yes | Disallowing Applebot-Extended keeps you in Apple search results while opting out of training |
| Meta | Meta-ExternalAgent / Meta-ExternalFetcher | The first crawls for AI training; the second fetches a specific page when a Meta AI user asks | The crawler does; the fetcher may bypass it for user-provided links | Excluded from Meta training; live fetches may still occur |
| Common Crawl | CCBot | Builds the open Common Crawl dataset that many AI labs have trained on | Yes | Excluded from that dataset |
Documentation changes. OpenAI revised its crawler page in 2025 to separate the search and training purposes more clearly; Anthropic split its agents into three in 2025. Re-read the source pages before you rely on a detail.
Three kinds of agent, three different decisions
Search indexers: allow them if you want to be found
OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and bingbot build the indexes that answer engines search. Blocking any of them removes you from that engine's answers. There is no privacy benefit for a business that publishes a website to be found.
User fetchers: mostly not your decision
ChatGPT-User, Perplexity-User, Claude-User and Meta-ExternalFetcher load a page because a person asked a question. Two of the four companies say plainly that robots.txt may not apply to these. What does stop them is a firewall challenge or a JavaScript-only page. Make sure your text is in the HTML.
Training crawlers: a real choice, and a separate one
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot collect content for model training. Every company listed states that blocking the training agent does not remove you from its search product. Publishers with valuable archives often block these. A local business whose goal is to be recommended generally allows them, because a model that has read your site in training is one more model that knows your name. Either answer is defensible. Make it on purpose.
A robots.txt that says yes on purpose
This is the shape of the file we publish on our own site and on client installs. Explicit allow lines do nothing that the default would not, but they document the decision, and they survive the next person who pastes in a "block all AI" snippet from a forum.
# Search and answer agents: allow User-agent: Googlebot Allow: / User-agent: bingbot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # Training crawlers: your call. This example allows them. User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / # Everyone else: normal WordPress rules User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.example.com/sitemap_index.xml
Where blocks hide
- Cloudflare and similar CDNs. Since July 1, 2025 Cloudflare asks new domains whether to allow AI crawlers and blocks them by default (Cloudflare press release, AI crawlers blocked by default for new domains (Jul 1, 2025)). Existing zones have a one-click "block AI bots" setting. Check the dashboard, not just robots.txt.
- Security plugins and WAFs. Bot-protection rules that challenge unknown user agents will challenge OAI-SearchBot too. Perplexity's documentation specifically asks WAF users to allow-list its agents (Perplexity, "Perplexity crawlers" (developer docs)).
- Website builder toggles. Some hosted platforms ship a site-level AI crawler switch. The default is not always "allow."
- Rendering. No block at all, but the page body arrives by script after the bot has left. The most common silent failure we find, and the one that cost our own site 17 points until we fixed it.
- Old robots.txt lines. A 2023 "Disallow: /" for GPTBot copied from a news article, forgotten ever since.
How to check in two minutes
- Open
yourdomain.com/robots.txtand search for each token above. - Fetch a page as a bot and read the response:
curl -A "OAI-SearchBot" https://yourdomain.com/should return your HTML, not a challenge page or a 403. - Look at the fetched HTML for your actual text. If the body is empty, you have a rendering problem, not a permissions problem.
- Check your CDN or firewall dashboard for an AI-bot setting.
- Or run the free scan, which does all of this and reads the schema while it is there.
Controls that limit AI use without leaving search
If you want to stay in search results but limit how much of a page an AI feature can reuse: Google honours nosnippet, data-nosnippet and max-snippet for AI Overviews and AI Mode because they use the same indexing (Google Search Central, "AI features and your website" (updated Dec 2025)); Bing honours nocache (URL, title and snippet only) and noarchive (excluded from Copilot answers and training) while keeping the page in Bing results (Bing Webmaster Blog, new options to control usage of content in Bing Chat (Sep 2023)). For a business that wants to be recommended, leave these off.
Straight answers about AI crawlers
Which AI crawlers should a local business allow?
All of the search and answer agents: Googlebot, bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User. Blocking any of them removes you from that engine's answers. Training crawlers such as GPTBot and ClaudeBot are a separate choice that does not affect search visibility.
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the agent that surfaces websites in ChatGPT search. Blocking GPTBot excludes your content from training; ChatGPT search uses OAI-SearchBot and ChatGPT-User.
Does Google-Extended control AI Overviews?
No. Google-Extended is a robots.txt token that governs whether content may be used for Gemini training and grounding. Google states it does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode draw on the regular Search index crawled by Googlebot.
Do all AI bots obey robots.txt?
The search indexers and training crawlers listed here say they do. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, in both cases because a person initiated the fetch.
Why does my site pass robots.txt but still not show up?
Usually one of three things: a CDN or firewall is challenging the bot, the page body only renders by JavaScript, or the business details on the site do not match the details elsewhere. The free scan checks all three.
Not sure what is blocking you?We check in fifteen minutes.
The free scan fetches your site the way each agent does, reads what came back, and tells you exactly which layer is in the way.
Get your free AI Visibility ScanBook a call- OpenAI, "Overview of OpenAI crawlers"
- Google Search Central, overview of Google crawlers and fetchers (Googlebot, Google-Extended)
- Google Search Central, "AI features and your website" (updated Dec 2025)
- Anthropic, "Does Anthropic crawl data from the web?" (Claude Help Center)
- Perplexity, "Perplexity crawlers" (developer docs)
- Bing Webmaster Blog, new options to control usage of content in Bing Chat (Sep 2023)
- Bing Webmaster Tools, robots meta tags Bing supports (nocache, noarchive)
- Apple Support, "About Applebot" (Applebot-Extended)
- Meta for Developers, Meta web crawlers
- Cloudflare press release, AI crawlers blocked by default for new domains (Jul 1, 2025)
DiamondBack scan scores measure what AI systems can read on a website (a 20-point technical check), not how often a business is cited. Third-party figures are quoted from the linked studies as published.