AI crawlers and robots.txt: which ones to let in
Many websites block AI crawlers without knowing it (theme, security plugin, host firewall). The result: ChatGPT or Perplexity can't read your pages, and recommend you less. Here's who's who.
By the PickableAI team · Updated 30 September 2026 · 2 min read
Two families of crawlers
Tell them apart:
- Search crawlers read your site to answer users' questions in real time. Blocking them means disappearing from those answers.
- Training crawlers collect text to train future models. Blocking them is a legitimate choice: it doesn't stop you being cited by search.
Who's who
| Crawler | Company | Role | Advice |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search | Allow |
ChatGPT-User | OpenAI | Visits a page at a user's request | Allow |
GPTBot | OpenAI | Model training | Your choice |
PerplexityBot | Perplexity | Search index | Allow |
Perplexity-User | Perplexity | Visits at a user's request | Allow |
Claude-SearchBot | Anthropic | Claude search | Allow |
Claude-User | Anthropic | Visits at a user's request | Allow |
ClaudeBot | Anthropic | Training | Your choice |
Googlebot | Google Search and Google's AI answers | Allow | |
Google-Extended | Use by Gemini (outside Google Search) | Your choice | |
Bingbot | Microsoft | Bing and Copilot (and services built on Bing) | Allow |
Applebot / Applebot-Extended | Apple | Siri search / training | Allow / your choice |
CCBot | Common Crawl | Open archive used to train models | Your choice |
The recommended robots.txt
For a local business that wants to be recommended, the simplest is to allow everything:
User-agent: * then Allow: /, plus a Sitemap: line with your sitemap's address.
If you'd rather refuse training while staying visible in answers, add separate User-agent: GPTBot, User-agent: ClaudeBot, User-agent: CCBot blocks with Disallow: /. Never block OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot or Bingbot if you want to be cited.
Hidden blocks
robots.txt isn't the only culprit:
- Firewalls and anti-bot protection (Cloudflare, WordPress security plugins): some block AI crawlers by default. Check the “AI bots” settings of your host or Cloudflare.
- Sites rendered with JavaScript: AI crawlers generally don't run JavaScript. If your text isn't in the HTML sent by the server, they see an empty page.
- A forgotten noindex tag after a redesign.
Our tool Your site as AI sees it checks these three points and your robots.txt in 20 seconds.
Guides by trade
- How to get your restaurant recommended by ChatGPT, Gemini and Perplexity
- How to get your garage recommended by ChatGPT, Gemini and Perplexity
- How to get your dental practice recommended by ChatGPT, Gemini and Perplexity
- How to get your hair salon recommended by ChatGPT, Gemini and Perplexity
- How to get your hotel recommended by ChatGPT, Gemini and Perplexity
Compare tools
Does AI recommend you?
The free check asks AI assistants your customers' questions (“best {trade} in {city}”) and shows your position, or who is named instead of you.
Check my business