Guide · AI search
AI crawlers and robots.txt: which bots do what and how to set each one
In short
AI companies run several bots with different jobs: search bots that fetch pages for answers with links, crawlers that collect training data, and fetchers that act when a user asks. Their owners document separate robots.txt tokens, for example OAI-SearchBot for ChatGPT search and GPTBot for training. robots.txt is a request, not access control. Whether to opt out of training is your decision; the vendors document the setting but do not recommend either choice.

Which bot does what, per its owner
| Bot | Purpose, as the owner describes it | Note |
|---|---|---|
| OAI-SearchBot (OpenAI) | Search in ChatGPT | Sites opted out will not be shown in ChatGPT search answers, though can still appear as navigational links; it can take about 24 hours to adjust |
| GPTBot (OpenAI) | Training generative AI foundation models | Disallowing it indicates content should not be used for training |
| ChatGPT-User (OpenAI) | Actions initiated by a user | Robots.txt rules may not apply |
| ClaudeBot (Anthropic) | Collecting web content that could contribute to training | Anthropic says its bots honour robots.txt directives |
| Claude-User, Claude-SearchBot (Anthropic) | User-directed retrieval; indexing for search | Disabling them may reduce visibility in search |
| PerplexityBot | Surface and link websites in Perplexity search | Not used to crawl content for AI foundation models; Perplexity-User generally ignores robots.txt |
| Google-Extended | Controls whether content Google crawls may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI | Does not affect inclusion in Google Search; no separate user-agent string |
| Applebot-Extended | Opt out of training generative models | Applebot may still crawl for search |
| CCBot (Common Crawl) | An open web archive | Block with User-agent: CCBot and Disallow: / |

Bot lists change often, so check each owner’s page before you edit your file. Microsoft’s Bing blog of 30 October 2026 adds a training preference that is set in robots.txt, and says an Allow setting does not override a robots.txt rule that blocks Bingbot from crawling.
How robots.txt works
RFC 9309, the standard, says the rules are not a form of access authorization, so they ask but do not enforce. The file must be at /robots.txt at the top of the site; the most specific matching rule wins; and if the file cannot be reached because of a server error, crawlers must assume a complete disallow, while if it is simply missing they may crawl anything. Crawlers should not normally use a cached copy for more than 24 hours. Matching of the bot name is case-insensitive.
Sample settings
These rules allow search bots and opt out of the training crawlers whose owners document a separate token. It is the combination OpenAI’s page describes, and your own decision, not a recommendation.
| User-agent | Rule | Effect |
|---|---|---|
| OAI-SearchBot | Allow: / | Can appear in ChatGPT search |
| GPTBot | Disallow: / | Asks OpenAI not to use pages for training |
| Claude-SearchBot | Allow: / | Can be indexed for search |
| ClaudeBot | Disallow: / | Asks Anthropic not to collect for training |
| PerplexityBot | Allow: / | Can appear in Perplexity results |
| Google-Extended | Disallow: / | Asks Google not to use pages for Gemini training or grounding |
| Applebot-Extended | Disallow: / | Opts out of Apple’s training use |
| CCBot | Disallow: / | Keeps Common Crawl out |
| * | Allow: / | Everyone else may crawl |
Google AI Overviews and AI Mode eligibility depends on Googlebot indexing and snippet controls, not on Google-Extended.
And llms.txt?
Google’s July 2026 guide says creating unnecessary AI text files such as llms.txt will neither harm nor help your visibility in Google Search, as Google Search ignores them. We found no official statement from Microsoft, OpenAI, Anthropic, Perplexity or Apple on whether their systems read it.
What Vokolla’s own file does
On 19 November 2026, vokolla.com/robots.txt allowed all crawlers on public pages and asked them to stay out of /api/ and /admin/ (public API paths stay open), and listed the site maps. No AI bot is named, so none is blocked. Our site also serves a /llms.txt describing our packages.
Publish with Vokolla
Vokolla shows one public price per release, from US$69, with no quote, sales call or subscription. You write the release or draft it with the Vokolla AI assistant, choose the package, pay online and get a report with every live link, and an editor reviews each release before it goes out. You can try it first with one free release every 30 days; every package is on the pricing page.
Questions people ask
What is the difference between OAI-SearchBot and GPTBot?
OAI-SearchBot is for search in ChatGPT; GPTBot is for training generative AI models, and each has its own robots.txt setting.
Does blocking GPTBot remove me from ChatGPT?
OpenAI says each setting is independent; sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.
Does Google-Extended affect Google Search?
Google says it does not impact inclusion in Google Search nor is it a ranking signal.
Is robots.txt a security measure?
No. RFC 9309 says its rules are not a form of access authorization.
Do I need an llms.txt file?
Google Search ignores it; other vendors have not said.
Vokolla pages for this step
- Packages →Six packages from US$69 to US$659 per release, each with a report of the live links.
- Pricing →Every package’s public price, reach and what it includes, side by side.
- Free press release →One free release every 30 days on news sites, after editor review.
- AI assistant →Connect your AI assistant to Vokolla to draft, check and save a release.
Sources
Written and checked by the Vokolla editorial team. Vokolla provides the packages described here; prices and outlets are as published on vokolla.com on the date above, and the results quoted come from our own releases. Search rankings depend on many factors, so no distribution service can guarantee them.
- OpenAI: Overview of OpenAI Crawlers (OAI-SearchBot, GPTBot, ChatGPT-User) — checked 2026-11-05
- Anthropic Help Center: Does Anthropic crawl data from the web? (ClaudeBot, Claude-User, Claude-SearchBot; honours robots.txt) — checked 2026-11-05
- Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User) — checked 2026-11-05
- Google Search Central: Google’s common crawlers (Google-Extended; updated 30 Oct 2026) — checked 2026-11-05
- Apple Support: About Applebot (Applebot-Extended opt-out from training; nosnippet) — checked 2026-11-05
- Common Crawl: CCBot (block with User-agent: CCBot, Disallow: /) — checked 2026-11-05
- Microsoft Bing Blogs: Your content, your choice: clearer preferences for training and grounding (30 Oct 2026; AI Training preference in robots.txt; an Allow setting does not override a robots.txt rule blocking Bingbot) — checked 2026-11-19
- IETF RFC 9309: Robots Exclusion Protocol (September 2022): not a form of access authorization; most specific match wins — checked 2026-11-05
- Google Search Central: Optimizing your website for generative AI features (updated 10 Jul 2026): no new files needed; llms.txt ignored; inauthentic mentions are not helpful — checked 2026-11-05
- Google Search Central: AI features and your website (updated 2025-12-10) — checked 2026-11-04
- Vokolla publishing records: our own releases on USA Today and partner networks (October 2026), Google, Bing and AI-answer checks on 4 and 7 Nov 2026 (internal records; the method is described on this page) — checked 2026-11-04
- Vokolla: Editorial guidelines for press releases — checked 2026-11-04
