Should I block AI crawlers from my website?

Answers · More Than Normal, Canterbury

The short answer

For most small businesses, blocking AI search crawlers is counterproductive because it can keep your business out of AI search answers. If you want to be found in assistants while opting out of model training, you can allow search crawlers such as OAI-SearchBot and block training crawlers such as GPTBot separately.

For most local businesses and professional practices, no, at least not across the board. Blocking every AI bot in robots.txt often blocks the very search crawlers that help AI assistants find and cite your business.

The useful distinction is between search crawlers and training crawlers, because the main AI companies give you separate controls for each.

Search crawlers vs. training crawlers

When a small business blocks “AI”, it often blocks two different jobs with one blunt rule.

  • Search crawlers index your public pages so an assistant can show and link to your business when someone asks a question. Examples include OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot and Perplexity’s PerplexityBot. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers (though they can still appear as navigational links), and Perplexity recommends allowing PerplexityBot if you want to appear.
  • Training crawlers and tokens govern whether your content can be used to train future AI models. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Google). Google notes that Google-Extended is a robots.txt token that controls whether content is used for Gemini model training and grounding, and does not affect standard Google Search inclusion or ranking.

OpenAI and Anthropic both let you allow their search crawlers while disallowing their training crawlers. That means you do not have to choose between total exposure and total invisibility.

When should you allow them, and when should you block?

In our view, the decision comes down to what is on your website and how you win work.

Allow search crawlers if your website exists to explain your services, prices, location and contact details to potential customers. Keeping OAI-SearchBot, Claude-SearchBot and PerplexityBot open supports the visibility work we describe in our answer on how to get your business mentioned by ChatGPT.

Block training crawlers only if you publish original research, paid-style guides or proprietary writing that you do not want used to train future models, while still wanting customers to find your business in AI search.

Block both if a section of your site holds private client material, internal documents or content you do not want indexed anywhere, though remember that robots.txt is voluntary and genuinely confidential pages belong behind a login rather than a robots.txt rule.

What can blocking in robots.txt not stop?

Two limits are worth knowing before you rely on robots.txt alone. First, user-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks an assistant to look something up live, and OpenAI notes that robots.txt rules may not apply to ChatGPT-User. Second, robots.txt is a set of instructions for crawlers that choose to follow it, not a lock on the door.

One final check is accidental blocking. Some hosting firewalls and security plugins can block automated bots by mistake, which can shut out search crawlers without your realising it. If you want AI assistants to find your business, check your robots.txt file and server firewall rules before assuming your content is being read.

Sources

Where a claim depends on how Google or another company says something works, we link to their own words so you can check it.

  1. Overview of OpenAI Crawlers, OpenAI. Supports that GPTBot (training) and OAI-SearchBot (ChatGPT search) can be controlled independently in robots.txt.
  2. What is ClaudeBot and the Claude crawlers, Anthropic. Supports the separate roles of ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-initiated fetches).
  3. PerplexityBot and Perplexity-User, Perplexity. Supports allowing PerplexityBot to appear in Perplexity search results and how Perplexity-User works.
  4. Google-Extended and Google crawlers, Google Search Central. Supports that Google-Extended controls Gemini model training and grounding without affecting standard Google Search visibility or ranking.

About this answer. We are an independent studio and this is our own research, experience and opinion, shared as general guidance. It is not legal, financial or other professional advice, and what works for one business may not work for another. Where the law or a regulator is involved, check the official guidance for your situation or ask a qualified adviser. Our sources are listed above.

Get our new answers by email

One short email a week with the answers we've published. Unsubscribe with one click.

We email you a link to confirm first. See our privacy notice.

Want to know where your website stands

A written SEO audit shows what to fix first, in order of priority, for £450.

Get an SEO auditBook a discovery call

No obligation, and we reply by email.