Cloudflare Crawler

GoodBotVerifiedAI CrawlerAGENT

Operated by Cloudflare Docs

What is Cloudflare Crawler?

Generated from Cloudflare, operator docs, and community data

Cloudflare Crawler is a web crawler operated by Cloudflare. The Cloudflare Crawler is a well-behaved crawler that retrieves web content. By default, it self-identifies as a bot, honors robots.txt directives, and cannot bypass CAPTCHAs or bot protection. Used by Cloudflare customers via the Browser Run /crawl endpoint. It is part of the growing wave of AI-powered crawlers that collect web data for training and improving AI models.

Helpful — Verified, safe crawler. Respects robots.txt and provides operator documentation.

Details

Bot Name
Cloudflare Crawler
Slug
cloudflare-browser-rendering-crawler
Kind
AGENT
Verification
Verified by Cloudflare
Category
AI Crawler
Operator
Cloudflare

User-Agent Patterns

CloudflareBrowserRenderingCrawler/1.0

Full User-Agent Strings

CloudflareBrowserRenderingCrawler/1.0

Main Use Cases

AI model training data collectionContent analysisWeb data extraction

Why It Crawls Your Site

Cloudflare Crawler crawls websites to collect and process web content for AI model training and improvement. It is operated by Cloudflare as part of their AI and machine learning infrastructure. If you see this bot in your server logs, it is a verified crawler and generally safe.

How to Block / Whitelist

Block Cloudflare Crawler

Add a Disallow rule for CloudflareBrowserRenderingCrawler/1.0 in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Cloudflare Crawler from crawling your entire site
User-agent: CloudflareBrowserRenderingCrawler/1.0
Disallow: /

# Allow Cloudflare Crawler full access
User-agent: CloudflareBrowserRenderingCrawler/1.0
Allow: /

Whitelist Cloudflare Crawler

Ensure your robots.txt allows CloudflareBrowserRenderingCrawler/1.0. Verify requests are genuine by checking the user-agent string and referring to Cloudflare's documentation.

Related Bots from Cloudflare

Other crawlers operated by Cloudflare

AI SearchCloudflare AI Search is a managed service that lets you connect your data and easily build AI-powered search.AI SearchBushbabyBushbaby is an internal bot used by Cloudflare. Its purpose is to manage and renew SSL certificates for websites that use Cloudflare's services.SecurityCloudflare Browser RunRenders web pages in headless browsers for Cloudflare customers. Used for browser automation (screenshots, PDF generation, content extraction, etc.) and for AI agents to interact with the web. Used by Cloudflare customers via Workers bindings and REST API. Does not include the /crawl endpoint, which has a separate bot identity (Cloudflare Crawler - Signed Agent).AI AssistantCloudflare CSUPCloudflare CSUP is a bot used by Cloudflare's customer support for diagnostic purposes. It is not a general web crawler and is used to investigate technical issues with customer websites. OtherCloudflare Custom Error Page CrawlerCloulflare internal service that crawls customer error pages in order to serve them directly from our edge network.SEOCloudflare CustomerSLO ProberA service to verify the identity of Cloudflare-internal Customer SLO probesMonitoring & AnalyticsCloudflare DiagnosticsCloudflare system bot that performs health checks and diagnostic testsMonitoring & AnalyticsCloudflare Digicert DCVCloudflare Digicert DCV service.SEO

Traffic & Trends

Cloudflare Radar Traffic

No Cloudflare traffic data available

Google Trends Interest

Links & References

Cloudflare Documentation

cloudflare.com

Cloudflare Radar: Cloudflare Crawler

radar.cloudflare.com/traffic/bots/cloudflare-browser-rendering-crawler

Data Sources

This profile was compiled from the following sources:

CloudflareRadar

Last updated: May 26, 2026