WE
WebObservationLab
GoodBotVerifiedBOTExperimental crawler collecting public web pages for a locally operated search index.
Identification
Bot name
WebObservationLab
Category
Search Engine Crawler
Kind
BOT
Verification
Verified bot
Cloudflare slug
webobservationlab
Rated Oct 24, 2026 from public evidence
How we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
Names itself honestly
Yes
Verified by Cloudflare
Yes
Says it follows robots.txt and opt-outs
Not stated
Ignores site rules
None found
Disguises itself or evades blocks
None found
Overloads sites
None found
How widely people complain
No complaint about this bot in the evidence
Reddit discussion
There is no Reddit discussion of this bot in the evidence
What we read
- AI Web Crawler Bots Gone Wild! e.g. ClaudeBot, DotBot, PetalBot 🤖 - hacking - Lucee Dev
- so AI crawlers just ignore your robots.txt and apologize when confronted lol : r/neocities
- robots.txt - Wikipedia
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- What Is Robots.txt? | How a Robots.txt File Works
Block or allow WebObservationLab
Block it
Add a Disallow rule for WebObservationLab in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block WebObservationLab from crawling your entire site User-agent: WebObservationLab Disallow: / # Allow WebObservationLab full access User-agent: WebObservationLab Allow: /
Allow & verify
Ensure your robots.txt allows WebObservationLab. Verify requests by checking the user-agent string.
Traffic & trends
Cloudflare Radar traffic
No Cloudflare traffic data available.
Google Trends interest
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
CloudflareRadar
Last updated October 25, 2026
Something wrong with this entry? Suggest a correction