cloudflare
GoodBotCommunity-sourcedA collection of automated agents operated by Cloudflare for infrastructure management, security, and performance optimization. Operated by Cloudflare, Inc..
What is cloudflare?
Cloudflare operates a suite of automated agents that perform critical functions across its global network, including security scanning, performance optimization, and infrastructure health checks. These bots are integral to the Cloudflare platform, ensuring that traffic is routed efficiently and protected from malicious actors. They operate transparently as part of the infrastructure layer, providing services like WAF (Web Application Firewall) updates and edge-caching validation.
Identification
Why it crawls your site
Cloudflare agents crawl or probe websites to verify origin server availability, update security rules, and ensure that cached content remains synchronized with the source. This activity is necessary to maintain the integrity and speed of the websites protected by their network. Site administrators can verify these requests by checking that the source IP addresses belong to Cloudflare's published IP ranges and by validating the request headers against Cloudflare's official documentation.
IP ranges
Requests from cloudflare originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow cloudflare
Add a Disallow rule for cloudflare in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block cloudflare from crawling your entire site User-agent: cloudflare Disallow: / # Allow cloudflare full access User-agent: cloudflare Allow: /
Ensure your robots.txt allows cloudflare. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Data sources
This profile is compiled from the following sources.