Directory/Categories/AI Search/oai-searchbot

oai-searchbot

Name: oai-searchbot
Author: OpenAI

GoodBotCommunity-sourced

A dedicated crawler used by OpenAI to index web content for the purpose of providing real-time, cited search results within ChatGPT. Operated by OpenAI.

GoodBot

Assessment

AI Search

What is oai-searchbot?

oai-searchbot is the official web crawler operated by OpenAI, designed to fetch and index public web pages to power the search capabilities of ChatGPT. Unlike general-purpose crawlers, its primary objective is to retrieve current information to ensure that AI responses are grounded in up-to-date, verifiable sources. It operates with a focus on transparency, allowing site owners to control its access via standard robots.txt directives.

Helpful — It is a verified crawler that respects robots.txt and provides attribution to content creators within AI-generated responses.

Identification

Bot name

oai-searchbot

Operator

OpenAI

Why it crawls your site

The bot crawls the web to build a comprehensive index of information that allows OpenAI's models to answer user queries about current events, technical documentation, and other time-sensitive topics. By processing this data, the bot enables the model to provide direct citations and links back to the original source material, fostering a symbiotic relationship between AI platforms and content publishers. Site administrators can verify the bot by performing a reverse DNS lookup on the originating IP address to ensure it resolves to the openai.com domain.

IP ranges

Requests from oai-searchbot originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 35 ranges

Block or allow oai-searchbot

Block it

Add a Disallow rule for oai-searchbot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block oai-searchbot from crawling your entire site
User-agent: oai-searchbot
Disallow: /

# Allow oai-searchbot full access
User-agent: oai-searchbot
Allow: /