AI crawlers and robots.txt: which ones to let in
Many websites block AI crawlers without knowing it (theme, security plugin, host firewall). The result: ChatGPT or Perplexity can't read your pages, and recommend you less. Here's who's who.
By the PickableAI team · Updated 30 September 2026 · 2 min read
Two families of crawlers
Tell them apart:
- Search crawlers read your site to answer users' questions in real time. Blocking them means disappearing from those answers.
- Training crawlers collect text to train future models. Blocking them is a legitimate choice: it doesn't stop you being cited by search.
Who's who
| Crawler | Company | Role | Advice |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search | Allow |
ChatGPT-User | OpenAI | Visits a page at a user's request | Allow |
GPTBot | OpenAI | Model training | Your choice |
PerplexityBot | Perplexity | Search index | Allow |
Perplexity-User | Perplexity | Visits at a user's request | Allow |
Claude-SearchBot | Anthropic | Claude search | Allow |
Claude-User | Anthropic | Visits at a user's request | Allow |
ClaudeBot | Anthropic | Training | Your choice |
Googlebot | Google Search and Google's AI answers | Allow | |
Google-Extended | Use by Gemini (outside Google Search) | Your choice | |
Bingbot | Microsoft | Bing and Copilot (and services built on Bing) | Allow |
Applebot / Applebot-Extended | Apple | Siri search / training | Allow / your choice |
CCBot | Common Crawl | Open archive used to train models | Your choice |
The recommended robots.txt
For a local business that wants to be recommended, the simplest is to allow everything:
User-agent: * then Allow: /, plus a Sitemap: line with your sitemap's address.
If you'd rather refuse training while staying visible in answers, add separate User-agent: GPTBot, User-agent: ClaudeBot, User-agent: CCBot blocks with Disallow: /. Never block OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot or Bingbot if you want to be cited.
Hidden blocks
robots.txt isn't the only culprit:
- Firewalls and anti-bot protection (Cloudflare, WordPress security plugins): some block AI crawlers by default. Check the “AI bots” settings of your host or Cloudflare.
- Sites rendered with JavaScript: AI crawlers generally don't run JavaScript. If your text isn't in the HTML sent by the server, they see an empty page.
- A forgotten noindex tag after a redesign.
Our tool Your site as AI sees it checks these three points and your robots.txt in 20 seconds.
Does AI recommend you?
The free check asks AI assistants your customers' questions (“best {trade} in {city}”) and shows your position, or who is named instead of you.
Check my business