SiteMonitor
UndecidedBotwhois.orgA cloud-based crawler used by Siteimprove to audit websites for quality assurance, SEO performance, accessibility compliance, and policy adherence. Operated by Siteimprove.
What is SiteMonitor?
SiteMonitor is the primary crawler operated by Siteimprove, a platform dedicated to digital experience management. It systematically traverses websites configured by Siteimprove customers to identify broken links, accessibility barriers, SEO optimization opportunities, and adherence to corporate content policies. By providing these automated insights, the bot enables organizations to maintain high standards of web governance and user experience.
Identification
How we rated this bot
We could not confirm that it identifies itself honestly.
What we read
- How to deal with AI crawlers that ignore robots.txt | Soups Ranjan posted on the topic | LinkedIn
- What Is Robots.txt? | How a Robots.txt File Works
- 20 Common Robots.txt Issues (and How to Avoid Them)
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- robots.txt configuration - Security - MDN Web Docs
Why it crawls your site
The bot crawls websites to gather the raw data necessary for Siteimprove's reporting dashboard, which alerts site administrators to technical or content-related issues. It operates exclusively on behalf of authorized users who have added their domains to the Siteimprove platform. You can verify the operator by checking that the bot respects robots.txt directives and originates from known Siteimprove IP ranges, ensuring it only accesses sites where the owner has explicitly requested monitoring services.
Block or allow SiteMonitor
Add a Disallow rule for Mozilla/5.0 (X11; Linux x86_64; compatible; SiteMonitor/1.0) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block SiteMonitor from crawling your entire site User-agent: Mozilla/5.0 (X11; Linux x86_64; compatible; SiteMonitor/1.0) Disallow: / # Allow SiteMonitor full access User-agent: Mozilla/5.0 (X11; Linux x86_64; compatible; SiteMonitor/1.0) Allow: /
Ensure your robots.txt allows Mozilla/5.0 (X11; Linux x86_64; compatible; SiteMonitor/1.0). Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction