Home › Guides › AI crawler access (robots.txt)
Is your site blocking AI crawlers? robots.txt for GPTBot, ClaudeBot & PerplexityBot
Before your brand can show up in ChatGPT, Perplexity, Claude or Gemini answers, those tools' crawlers have to be able to reach your pages. One stray line in robots.txt can quietly make you invisible to AI search — and most teams never check. Here's exactly which bots matter, which are safe to block, and how to audit it in two minutes.
Two kinds of AI crawler — don't confuse them
AI companies run separate crawlers for two different jobs, and you can allow one while blocking the other:
- Live retrieval / search bots — fetch pages in real time to answer a user's question and cite sources. Blocking these means you can't appear in AI answers. The key ones:
OAI-SearchBot&ChatGPT-User(OpenAI),Claude-SearchBot(Anthropic),PerplexityBot&Perplexity-User, plusGooglebot/Bingbotfor Google AI Overviews and Copilot. - Training crawlers — collect data to train foundation models. Blocking these is a legitimate content/privacy choice and does not stop you appearing in live answers:
GPTBot,ClaudeBot,CCBot(Common Crawl),Google-Extended,Applebot-Extended.
The costly mistake: blanket-blocking everything "AI" (often copied from a template) and accidentally killing your retrieval visibility along with training.
A sensible 2026 robots.txt for AI search
A common, balanced setup — allow the search/citation bots, decide on training per your policy:
# Let AI search engines find and cite you
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# (Optional) opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /Always keep your normal search engines (Googlebot, Bingbot) allowed — they feed Google AI Overviews and Copilot too.
How to check what you're blocking
Open yoursite.com/robots.txt and look for any Disallow: / under the bot names above — or, faster, run an automated audit. The GEO Auditor checks your robots.txt against 8 AI search crawlers, flags any that are blocked, and scores the rest of your AI-readiness (structured data, content extractability, speed) — with a prioritized fix list.
Two-minute check. Enter your domain and see instantly whether AI crawlers can reach you — plus what else is holding back your AI-search visibility.
Run GEO Auditor on Apify →Allowed the bots — now what?
Crawler access is necessary but not sufficient. Once AI engines can read you, the next questions are whether they actually mention you and which sources they cite. Track that with AI Brand Monitor — it measures your visibility and share-of-voice across ChatGPT, Gemini and Claude, so you can see crawler-access fixes turn into real answer presence.
FAQ
Will blocking GPTBot remove me from ChatGPT? No — GPTBot is training. ChatGPT's live answers use OAI-SearchBot / ChatGPT-User; keep those allowed.
Is reading robots.txt enough? It's the first gate. After access, structured data, fast SSR pages and extractable content decide whether you're actually cited — all covered in the audit.