Cloudflare Robots API
GoodBotVerifiedBOTFetches robots.txt files for Cloudflare AI Crawl Control to retrieve and evaluate crawler directives. Operated by Cloudflare.
What is Cloudflare Robots API?
The Cloudflare Robots API is an internal tool designed to assist website administrators in verifying how their infrastructure handles automated traffic. By utilizing the /crawl endpoint, it allows users to simulate how different bots interact with their site, ensuring that security rules, WAF policies, and AI crawl control settings are functioning as intended. It serves as a diagnostic utility within the Cloudflare ecosystem to provide transparency into how content is being accessed by external agents.
Identification
CloudflareRobotsAPIHow we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
What we read
- Cloudflare goes after Google's AI Overviews with a new license for 20% of the web | Business Insider Africa
- and Google. By telling these "robots" where not to go on your site,
- and Google. By telling these "robots" where not to go on your site,
- APIs Explained (in 4 Minutes) - YouTube
- Stevey's Google Platforms Rant - Discover gists - GitHub
Why it crawls your site
This bot crawls websites to provide site owners with actionable data regarding their security posture and compliance with robots.txt directives. It acts as a verification mechanism, allowing users to confirm that their chosen blocking or allowing policies are correctly applied to incoming traffic. You can verify this bot by checking that the requests originate from Cloudflare's verified infrastructure and match the expected User-Agent signature.
Block or allow Cloudflare Robots API
Add a Disallow rule for CloudflareRobotsAPI in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Cloudflare Robots API from crawling your entire site User-agent: CloudflareRobotsAPI Disallow: / # Allow Cloudflare Robots API full access User-agent: CloudflareRobotsAPI Allow: /
Ensure your robots.txt allows CloudflareRobotsAPI. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Related bots
Other crawlers operated by Cloudflare.
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction