The short answer
For most small businesses, blocking AI search crawlers is counterproductive because it can keep your business out of AI search answers. If you want to be found in assistants while opting out of model training, you can allow search crawlers such as OAI-SearchBot and block training crawlers such as GPTBot separately.
For most local businesses and professional practices, no, at least not across the board. Blocking every AI bot in robots.txt often blocks the very search crawlers that help AI assistants find and cite your business.
The useful distinction is between search crawlers and training crawlers, because the main AI companies give you separate controls for each.
Search crawlers vs. training crawlers
When a small business blocks “AI”, it often blocks two different jobs with one blunt rule.
- Search crawlers index your public pages so an assistant can show and link to your business when someone asks a question. Examples include OpenAI’s
OAI-SearchBot, Anthropic’sClaude-SearchBotand Perplexity’sPerplexityBot. OpenAI states that sites opted out ofOAI-SearchBotwill not be shown in ChatGPT search answers (though they can still appear as navigational links), and Perplexity recommends allowingPerplexityBotif you want to appear. - Training crawlers and tokens govern whether your content can be used to train future AI models. Examples include
GPTBot(OpenAI),ClaudeBot(Anthropic) andGoogle-Extended(Google). Google notes thatGoogle-Extendedis arobots.txttoken that controls whether content is used for Gemini model training and grounding, and does not affect standard Google Search inclusion or ranking.
OpenAI and Anthropic both let you allow their search crawlers while disallowing their training crawlers. That means you do not have to choose between total exposure and total invisibility.
When should you allow them, and when should you block?
In our view, the decision comes down to what is on your website and how you win work.
Allow search crawlers if your website exists to explain your services, prices, location and contact details to potential customers. Keeping OAI-SearchBot, Claude-SearchBot and PerplexityBot open supports the visibility work we describe in our answer on how to get your business mentioned by ChatGPT.
Block training crawlers only if you publish original research, paid-style guides or proprietary writing that you do not want used to train future models, while still wanting customers to find your business in AI search.
Block both if a section of your site holds private client material, internal documents or content you do not want indexed anywhere, though remember that robots.txt is voluntary and genuinely confidential pages belong behind a login rather than a robots.txt rule.
What can blocking in robots.txt not stop?
Two limits are worth knowing before you rely on robots.txt alone. First, user-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks an assistant to look something up live, and OpenAI notes that robots.txt rules may not apply to ChatGPT-User. Second, robots.txt is a set of instructions for crawlers that choose to follow it, not a lock on the door.
One final check is accidental blocking. Some hosting firewalls and security plugins can block automated bots by mistake, which can shut out search crawlers without your realising it. If you want AI assistants to find your business, check your robots.txt file and server firewall rules before assuming your content is being read.
Sources
Where a claim depends on how Google or another company says something works, we link to their own words so you can check it.
- Overview of OpenAI Crawlers, OpenAI. Supports that GPTBot (training) and OAI-SearchBot (ChatGPT search) can be controlled independently in robots.txt.
- What is ClaudeBot and the Claude crawlers, Anthropic. Supports the separate roles of ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-initiated fetches).
- PerplexityBot and Perplexity-User, Perplexity. Supports allowing PerplexityBot to appear in Perplexity search results and how Perplexity-User works.
- Google-Extended and Google crawlers, Google Search Central. Supports that Google-Extended controls Gemini model training and grounding without affecting standard Google Search visibility or ranking.
About this answer. We are an independent studio and this is our own research, experience and opinion, shared as general guidance. It is not legal, financial or other professional advice, and what works for one business may not work for another. Where the law or a regulator is involved, check the official guidance for your situation or ask a qualified adviser. Our sources are listed above.