CR
CrawlCheck
GoodBotVerifiedBOTSite audit tool: on a user's request, fetches a site's public pages and reports what search and AI crawlers are served.
Identification
Bot name
CrawlCheck
Category
SEO
Kind
BOT
Verification
Verified bot
Cloudflare slug
crawlcheck
Rated Oct 24, 2026 from public evidence
How we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
Names itself honestly
Yes
Verified by Cloudflare
Yes
Says it follows robots.txt and opt-outs
Not stated
Ignores site rules
None found
Disguises itself or evades blocks
None found
Overloads sites
None found
How widely people complain
No complaint about this bot in the evidence
Reddit discussion
There is no Reddit discussion of this bot in the evidence
What we read
- 🐛 [Bug/Feature]: Not respecting robots.txt · Issue #275 · coleam00/Archon · GitHub
- I wonder if it will actually respect the robots.txt this time. | Hacker News
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- Robots.txt isn't preventing my site from being crawled
- robots.txt - Wikipedia
Block or allow CrawlCheck
Block it
Add a Disallow rule for CrawlCheck in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block CrawlCheck from crawling your entire site User-agent: CrawlCheck Disallow: / # Allow CrawlCheck full access User-agent: CrawlCheck Allow: /
Allow & verify
Ensure your robots.txt allows CrawlCheck. Verify requests by checking the user-agent string.
Traffic & trends
Cloudflare Radar traffic
No Cloudflare traffic data available.
Google Trends interest
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
CloudflareRadar
Last updated October 25, 2026
Something wrong with this entry? Suggest a correction