AI crawler

An AI crawler is a bot that an AI company uses to fetch web pages. OpenAI, Anthropic and Perplexity each run several, with different jobs, and each bot can be allowed or blocked on its own in robots.txt. They fall into three groups:

Why it matters for AI visibility

Blocking a search bot can take you out of that engine's answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and Anthropic says that disabling Claude-SearchBot prevents its system from indexing your content for search, which may reduce your visibility in Claude's results. Training bots are a separate choice. OpenAI states that each of its settings is independent, so a site can block GPTBot and still allow OAI-SearchBot.

How to manage them

• robots.txt: write a group for each bot you want to treat differently. Crawlers are asked to honor these rules, but RFC 9309 says they "are not a form of access authorization."
• User-triggered fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
• CDN and firewall: these can block bots that robots.txt allows. Cloudflare, for example, lets sites set separate policies for search, agent and training traffic.
• Logs: count requests by user agent, and verify IP addresses against the lists vendors publish.

Google works differently. Googlebot crawls for Google Search, including AI Overviews and AI Mode, and Google-Extended is a robots.txt token rather than a separate crawler. For a full audit, see the brand visibility checklist for AI platforms.