New: publish your first press release free on up to 5 news sites, reviewed by our editors. Publish a release free

AI crawlers and robots.txt: which bots do what and how to set each one

In short

AI companies run several bots with different jobs: search bots that fetch pages for answers with links, crawlers that collect training data, and fetchers that act when a user asks. Their owners document separate robots.txt tokens, for example OAI-SearchBot for ChatGPT search and GPTBot for training. robots.txt is a request, not access control. Whether to opt out of training is your decision; the vendors document the setting but do not recommend either choice.

AI crawlers and robots.txt. 3 jobs: search, training, and fetches a user starts. ~24 h OpenAI says search settings can take about a day to adjust. Not a lock RFC 9309: the rules are not a form of access authorization. Ignored Google Search ignores llms.txt (July 2026 guide). Allow search bots; opting out of training is your own decision.
AI crawlers and robots.txt at a glance.

Which bot does what, per its owner

As documented by each companyScroll sideways to see the whole table.
BotPurpose, as the owner describes itNote
OAI-SearchBot (OpenAI)Search in ChatGPTSites opted out will not be shown in ChatGPT search answers, though can still appear as navigational links; it can take about 24 hours to adjust
GPTBot (OpenAI)Training generative AI foundation modelsDisallowing it indicates content should not be used for training
ChatGPT-User (OpenAI)Actions initiated by a userRobots.txt rules may not apply
ClaudeBot (Anthropic)Collecting web content that could contribute to trainingAnthropic says its bots honour robots.txt directives
Claude-User, Claude-SearchBot (Anthropic)User-directed retrieval; indexing for searchDisabling them may reduce visibility in search
PerplexityBotSurface and link websites in Perplexity searchNot used to crawl content for AI foundation models; Perplexity-User generally ignores robots.txt
Google-ExtendedControls whether content Google crawls may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AIDoes not affect inclusion in Google Search; no separate user-agent string
Applebot-ExtendedOpt out of training generative modelsApplebot may still crawl for search
CCBot (Common Crawl)An open web archiveBlock with User-agent: CCBot and Disallow: /
Four bots to know. Bot: OAI-SearchBot, Job: Search in ChatGPT, Note: Opt-out: not shown in answers; Bot: GPTBot, Job: Training, Note: Separate setting; Bot: ChatGPT-User, Job: Fetch a user asked for, Note: robots.txt may not apply; Bot: Google-Extended, Job: Gemini training and grounding control, Note: Not part of Google Search
Four AI bots and their jobs.

Bot lists change often, so check each owner’s page before you edit your file. Microsoft’s Bing blog of 30 October 2026 adds a training preference that is set in robots.txt, and says an Allow setting does not override a robots.txt rule that blocks Bingbot from crawling.

How robots.txt works

RFC 9309, the standard, says the rules are not a form of access authorization, so they ask but do not enforce. The file must be at /robots.txt at the top of the site; the most specific matching rule wins; and if the file cannot be reached because of a server error, crawlers must assume a complete disallow, while if it is simply missing they may crawl anything. Crawlers should not normally use a cached copy for more than 24 hours. Matching of the bot name is case-insensitive.

Sample settings

These rules allow search bots and opt out of the training crawlers whose owners document a separate token. It is the combination OpenAI’s page describes, and your own decision, not a recommendation.

Example rules (each is a User-agent line with the rule that follows)Scroll sideways to see the whole table.
User-agentRuleEffect
OAI-SearchBotAllow: /Can appear in ChatGPT search
GPTBotDisallow: /Asks OpenAI not to use pages for training
Claude-SearchBotAllow: /Can be indexed for search
ClaudeBotDisallow: /Asks Anthropic not to collect for training
PerplexityBotAllow: /Can appear in Perplexity results
Google-ExtendedDisallow: /Asks Google not to use pages for Gemini training or grounding
Applebot-ExtendedDisallow: /Opts out of Apple’s training use
CCBotDisallow: /Keeps Common Crawl out
*Allow: /Everyone else may crawl

Google AI Overviews and AI Mode eligibility depends on Googlebot indexing and snippet controls, not on Google-Extended.

And llms.txt?

Google’s July 2026 guide says creating unnecessary AI text files such as llms.txt will neither harm nor help your visibility in Google Search, as Google Search ignores them. We found no official statement from Microsoft, OpenAI, Anthropic, Perplexity or Apple on whether their systems read it.

What Vokolla’s own file does

On 19 November 2026, vokolla.com/robots.txt allowed all crawlers on public pages and asked them to stay out of /api/ and /admin/ (public API paths stay open), and listed the site maps. No AI bot is named, so none is blocked. Our site also serves a /llms.txt describing our packages.

Publish with Vokolla

Vokolla shows one public price per release, from US$69, with no quote, sales call or subscription. You write the release or draft it with the Vokolla AI assistant, choose the package, pay online and get a report with every live link, and an editor reviews each release before it goes out. You can try it first with one free release every 30 days; every package is on the pricing page.

Questions people ask

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot is for search in ChatGPT; GPTBot is for training generative AI models, and each has its own robots.txt setting.

Does blocking GPTBot remove me from ChatGPT?

OpenAI says each setting is independent; sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.

Does Google-Extended affect Google Search?

Google says it does not impact inclusion in Google Search nor is it a ranking signal.

Is robots.txt a security measure?

No. RFC 9309 says its rules are not a form of access authorization.

Do I need an llms.txt file?

Google Search ignores it; other vendors have not said.

Vokolla pages for this step

  • Packages →Six packages from US$69 to US$659 per release, each with a report of the live links.
  • Pricing →Every package’s public price, reach and what it includes, side by side.
  • Free press release →One free release every 30 days on news sites, after editor review.
  • AI assistant →Connect your AI assistant to Vokolla to draft, check and save a release.

Sources

Written and checked by the Vokolla editorial team. Vokolla provides the packages described here; prices and outlets are as published on vokolla.com on the date above, and the results quoted come from our own releases. Search rankings depend on many factors, so no distribution service can guarantee them.