The short answer
OpenAI uses three main bots for separate jobs. GPTBot crawls pages that may be used to train AI models, OAI-SearchBot indexes sites for ChatGPT search results, and ChatGPT-User fetches a page when a person asks ChatGPT to look something up. You can allow search without allowing training.
They are three different OpenAI bots, each built for a separate job. OpenAI’s crawler documentation separates them so a website owner can appear in ChatGPT search without opting into AI model training.
How do the three bots compare?
OpenAI describes each bot’s purpose on its crawler overview page (alongside a fourth bot, OAI-AdsBot, which only checks landing pages submitted as ads in ChatGPT).
GPTBotis for model training. It crawls public web pages to collect content that may be used to train OpenAI’s generative AI foundation models. DisallowingGPTBottells OpenAI not to use your content for training.OAI-SearchBotis for ChatGPT search. It crawls pages to surface websites in ChatGPT’s search results. OpenAI says sites that opt out ofOAI-SearchBotwill not be shown in ChatGPT search answers, though they can still appear as navigational links.ChatGPT-Useris for live user requests. It is not an automated background crawler. OpenAI usesChatGPT-Userwhen a person asks ChatGPT, a Custom GPT or an external GPT Action to visit a page during a conversation.
OpenAI treats GPTBot and OAI-SearchBot as independent settings. You can block GPTBot and allow OAI-SearchBot, or allow both. If both are allowed, OpenAI says it may use a single crawl for both purposes to avoid hitting your server twice.
How does robots.txt treat each one?
A robots.txt file sits at the root of your site and gives automated crawlers instructions on which addresses they may visit. As Google’s robots.txt guide notes, if you use a website builder such as Wix or Blogger, you may not need to edit the file directly because the platform may provide search visibility settings instead.
OpenAI says robots.txt tags control GPTBot and OAI-SearchBot independently, and that it can take around 24 hours after a robots.txt update for its search systems to adjust.
ChatGPT-User is different. Because it visits a page only when a human user triggers a request, OpenAI states that robots.txt rules may not apply to it, and it does not decide whether your site appears in ChatGPT search. Google’s specification page draws a similar line for Google’s own crawlers, noting that its automated crawlers support the Robots Exclusion Protocol while the protocol is not applicable to user-controlled crawlers such as feed subscriptions.
Which ones should a small business allow?
In our view, most local businesses and professional practices want OAI-SearchBot allowed. Blocking it keeps your pages out of ChatGPT search answers, which is the opposite of what you want if customers ask ChatGPT for recommendations in your area. Our answer on how to get your business mentioned by ChatGPT covers the wider steps that help.
Whether to allow or block GPTBot is a separate choice. If your site holds public service pages, prices and contact details, allowing GPTBot carries little obvious downside. If you publish original guides or research that you would rather not contribute to model training, disallowing GPTBot while keeping OAI-SearchBot open is the cleanest split.
Two practical limits are worth keeping in mind. First, robots.txt is voluntary and does not stop a page from being listed if another site links to it, so Google advises using password protection or noindex to hide genuinely private pages. Second, if a security firewall is blocking OpenAI’s bots by mistake, OpenAI publishes official JSON IP lists on its crawler page so a developer or host can verify genuine OpenAI traffic.
Sources
Where a claim depends on how Google or another company says something works, we link to their own words so you can check it.
- Overview of OpenAI Crawlers, OpenAI. Supports what GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot do, independent robots.txt control, the 24-hour adjustment window and published IP ranges.
- Introduction to robots.txt, Google Search Central. Supports what a robots.txt file is for, how website builders such as Wix and Blogger handle it and why it does not hide a linked page on its own.
- How Google Interprets the robots.txt Specification, Google Search Central. Supports that automated crawlers follow the Robots Exclusion Protocol while user-controlled fetchers may not.
About this answer. We are an independent studio and this is our own research, experience and opinion, shared as general guidance. It is not legal, financial or other professional advice, and what works for one business may not work for another. Where the law or a regulator is involved, check the official guidance for your situation or ask a qualified adviser. Our sources are listed above.