gptbot
GoodBotCommunity-sourcedGPTBot is the official web crawler used by OpenAI to collect publicly available data from the internet to improve the performance, accuracy, and safety of their AI models. Operated by OpenAI.
What is gptbot?
GPTBot is the primary mechanism OpenAI uses to ingest web content for the training and refinement of its generative AI systems, including GPT-4 and future iterations. By systematically traversing the web, it gathers diverse datasets that help the models understand human language, context, and factual information. OpenAI maintains this bot as a transparent way for site owners to manage their content's inclusion in AI training pipelines.
Identification
Why it crawls your site
The bot crawls the web to build a comprehensive knowledge base that enables OpenAI's models to provide more relevant and accurate responses to user queries. It processes text, code, and other structured data to identify patterns and improve model alignment. Site owners can verify the bot's authenticity by checking that the reverse DNS lookup of the requesting IP address resolves to an openai.com domain, and they can restrict its access by updating their site's robots.txt file.
IP ranges
Requests from gptbot originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow gptbot
Add a Disallow rule for gptbot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block gptbot from crawling your entire site User-agent: gptbot Disallow: / # Allow gptbot full access User-agent: gptbot Allow: /
Ensure your robots.txt allows gptbot. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Related bots
Other crawlers operated by OpenAI.
Data sources
This profile is compiled from the following sources.