Cloudflare Robots API

The Cloudflare Robots API is an internal tool designed to assist website administrators in verifying how their infrastructure handles automated traffic. By utilizing the /crawl endpoint, it allows users to simulate how different bots interact with their site, ensuring that security rules, WAF policies, and AI crawl control settings are functioning as intended. It serves as a diagnostic utility within the Cloudflare ecosystem to provide transparency into how content is being accessed by external agents.

This bot crawls websites to provide site owners with actionable data regarding their security posture and compliance with robots.txt directives. It acts as a verification mechanism, allowing users to confirm that their chosen blocking or allowing policies are correctly applied to incoming traffic. You can verify this bot by checking that the requests originate from Cloudflare's verified infrastructure and match the expected User-Agent signature.