claudebot
GoodBotCommunity-sourcedA dedicated crawler operated by Anthropic used to index web content for the training and improvement of Claude AI models. Operated by Anthropic.
What is claudebot?
claudebot is the official web crawler deployed by Anthropic to gather data from the public internet. This data is processed and utilized to enhance the capabilities, safety, and knowledge breadth of the Claude family of large language models. By systematically visiting websites, the bot ensures that Anthropic's models remain informed by current information and diverse human-generated content.
Identification
Why it crawls your site
The primary purpose of claudebot is to ingest high-quality, publicly available web data to refine the training datasets used for Anthropic's AI systems. Anthropic maintains transparency regarding this activity by adhering to standard web protocols, including honoring robots.txt instructions that disallow crawling. Site owners can verify the bot's legitimacy by checking the official Anthropic documentation, which outlines how to manage or block the crawler's access to their domains.
IP ranges
Requests from claudebot originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow claudebot
Add a Disallow rule for claudebot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block claudebot from crawling your entire site User-agent: claudebot Disallow: / # Allow claudebot full access User-agent: claudebot Allow: /
Ensure your robots.txt allows claudebot. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Data sources
This profile is compiled from the following sources.