OAI-SearchBot

OAI-SearchBot is the crawler OpenAI uses to surface websites in search results in ChatGPT's search features. According to OpenAI's crawler documentation, sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

How it differs from OpenAI's other bots

OpenAI documents four agents with separate jobs, and each setting is independent of the others:

• OAI-SearchBot finds pages for ChatGPT's search features.
• GPTBot collects content to make OpenAI's foundation models more useful and safe. Disallowing it signals that your content should not be used to train them.
• ChatGPT-User fetches pages for some user actions in ChatGPT and custom GPTs. It does not crawl the web automatically, and because a user starts the request, OpenAI says robots.txt rules may not apply.
• OAI-AdsBot only visits pages submitted as ads on ChatGPT.

So a site can keep its content out of training by blocking GPTBot and still appear in ChatGPT search by allowing OAI-SearchBot:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

How to check it

• robots.txt: look for a group that names OAI-SearchBot. If there is none, the User-agent: * group applies, as RFC 9309 requires.
• CDN and firewall: OpenAI recommends allowing requests from the IP ranges it publishes at openai.com/searchbot.json.
• Logs: filter for the OAI-SearchBot user agent and confirm its requests get 200 responses, not blocks or challenges.
• Timing: OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

Then check the answers. Run your prompts in ChatGPT with search and see whether your pages appear as citations. The other bots that matter for AI answers are covered under AI crawler.