Should I block AI crawlers?
If you want AI assistants to recommend your business, allow their search and assistant crawlers. Blocking the training crawlers is a separate choice: it keeps your pages out of model training, and it can reduce what the models know about you. Check your CDN and firewall as well as robots.txt, because many sites block all of them without meaning to.
By Gavin Levine, Founder and Principal Consultant
The three kinds of AI crawler
Each AI company runs more than one crawler, and each has its own name in robots.txt. A search crawler builds the index an assistant searches when it answers a question. An assistant crawler fetches a page at the moment a person asks about it. A training crawler collects text used to train future models.
Blocking one kind does not block the others. That is why "should I block AI?" has two answers.
What to allow if you want to be recommended
- The search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot, alongside Googlebot and Bingbot
- The assistant crawlers: ChatGPT-User, Claude-User and Perplexity-User, which fetch your page when someone asks about you
- Googlebot and Bingbot in every case. Google’s AI answers are built on Google’s index, and several assistants lean on Bing’s
When blocking training crawlers makes sense
Block them when your content is what you sell: journalism, paid research, courses, original writing or images. Allow them when your pages exist to be repeated, which is true of most local business websites. Your services, prices, hours and location are facts you want every model to have.
You can split it by section. A software company can let training crawlers read its marketing pages and keep them off the pages that describe how the product works.
How sites block AI crawlers by accident
robots.txt is not the only place a crawler can be refused, so a site can block all of them without anyone choosing to.
| Company | Search | Assistant, on request | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | None named |
| Googlebot | Googlebot | Google-Extended (a robots.txt control, not a separate crawler) | |
| Apple | Applebot | Applebot | Applebot-Extended (a robots.txt control, not a separate crawler) |
| Common Crawl | None | None | CCBot, an open archive many models are trained on |
Questions people ask
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended controls whether your content is used for Gemini and does not affect Google Search or how you rank. Google’s AI answers in Search are governed by Googlebot.
Do AI crawlers obey robots.txt?
The crawlers named here are documented by their companies as following it. robots.txt is still a request, so a crawler that hides its identity can ignore it. Enforcement needs a firewall rule.
Can I allow AI search and still block training?
Yes. Allow the search and assistant crawlers and disallow the training ones. They have separate names so that you can make that choice.
See what AI crawlers find on your site
The scan checks whether ChatGPT, Claude, Perplexity and Google can get into your site, read it and tell who you are. You keep the results either way.