Cerebro
GoodBotVerifiedAGENTAssistant used by many people. Each visit is one user's confirmed task, up to checkout. Signed, never disguised. Operated by Cerebro.
What is Cerebro?
Cerebro is a specialized web agent designed to assist users in completing specific online tasks, such as purchasing tickets or booking appointments. Unlike traditional crawlers that index content for search or model training, Cerebro operates on a per-task basis for a single authenticated user. It mimics human interaction patterns and strictly adheres to site-specific access controls, ensuring it only visits pages necessary to fulfill the user's immediate request.
Identification
How we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
What we read
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- 9 Robots.txt Rules Googlebot Ignores Even If You Add Them to Your File
- Google Says Why It May Ignore Robots.txt And Negatively Impact SEO
- How Google Interprets the robots.txt Specification | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers
- What Is a robots.txt File? (And When To Use One)
Why it crawls your site
Cerebro does not crawl or scrape websites in the traditional sense; it visits specific pages only when triggered by a user to perform a functional action. It identifies itself using HTTP Message Signatures (RFC 9421) and a consistent User-Agent string to maintain transparency with site administrators. The bot is designed to be respectful of site policies, and it will cease activity immediately if a site's bot protection or robots.txt configuration denies access.
Block or allow Cerebro
Add a Disallow rule for Cerebro in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Cerebro from crawling your entire site User-agent: Cerebro Disallow: / # Allow Cerebro full access User-agent: Cerebro Allow: /
Ensure your robots.txt allows Cerebro. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction