Bot WHOIS
Submit a botLook up a bot
Directory/Categories/Search Engine Crawler/Cloudflare Web Scanner
Cloudflare, Inc. logo

Cloudflare Web Scanner

GoodBotVerifiedBOT

Cloudflare Web Scanner is a well-behaved crawler that checks domain liveness, content type, robots.txt directives, and other web properties to map the shape of the web. Operated by Cloudflare, Inc..

GoodBot
Assessment
Search Engine Crawler
Category
BOT
Kind
1
Data sources
AI-generated summary, drawn from Cloudflare, operator docs, and community data

What is Cloudflare Web Scanner?

Cloudflare Web Scanner is an internal diagnostic tool used by the Cloudflare platform to help customers gain visibility into how automated agents and AI crawlers interact with their web properties. Unlike third-party scrapers, this bot operates within the Cloudflare ecosystem to provide site owners with actionable data regarding their security posture and crawler access policies. It is designed to assist in the enforcement of robots.txt directives and to help users manage the balance between content accessibility and protection against unauthorized data harvesting.

Helpful — This bot is a diagnostic tool provided by Cloudflare to help site owners audit their own security configurations and monitor AI crawler behavior.

Identification

Bot name
Cloudflare Web Scanner
Operator
Cloudflare, Inc.
Category
Search Engine Crawler
Kind
BOT
Verification
Verified bot
Cloudflare slug
cloudflare-web-scanner
User-agent patterns
CloudflareWebScanner
User-agent string
CloudflareWebScanner/1.0
Main use cases
Robots.txt compliance auditingAI crawler traffic monitoringSecurity configuration validation
Rated Sep 23, 2026 from public evidence

How we rated this bot

Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.

Names itself honestly
Yes
Verified by Cloudflare
Yes
Says it follows robots.txt and opt-outs
Not stated
Ignores site rules
None found
Disguises itself or evades blocks
None found
Overloads sites
None found
How widely people complain
No complaint about this bot in the evidence
Reddit discussion
There is no Reddit discussion of this bot in the evidence

What we read

Why it crawls your site

The bot crawls websites to provide Cloudflare customers with a comprehensive audit of their site's exposure to AI and automated traffic. By scanning pages, it helps verify that a site's security rules, such as WAF policies and robots.txt instructions, are being correctly interpreted and respected by external entities. This allows administrators to identify which AI services are accessing their content and to implement granular control policies through the Cloudflare dashboard.

Block or allow Cloudflare Web Scanner

Block it

Add a Disallow rule for CloudflareWebScanner in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Cloudflare Web Scanner from crawling your entire site
User-agent: CloudflareWebScanner
Disallow: /

# Allow Cloudflare Web Scanner full access
User-agent: CloudflareWebScanner
Allow: /
Allow & verify

Ensure your robots.txt allows CloudflareWebScanner. Verify requests by checking the user-agent string.

Traffic & trends

Cloudflare Radar traffic

No Cloudflare traffic data available.

Google Trends interest

Loading trend data…

Related bots

Other crawlers operated by Cloudflare, Inc..

CL
cloudflare
Verified Bot
›

Links & references

Cloudflare, Inc. documentation
developers.cloudflare.com/ai-crawl-control
↗
Cloudflare Radar: Cloudflare Web Scanner
radar.cloudflare.com/traffic/bots/cloudflare-web-scanner
↗

Data sources

This profile is compiled from the following sources.

CloudflareRadar
Last updated October 25, 2026

Something wrong with this entry? Suggest a correction