DotBot/1.2
UndecidedCommunity-sourcedDotBot is a web crawler developed by Moz, designed to efficiently and responsibly collect data from websites. It is utilized by Moz's tools, such as Open Site Explorer, to gather information for SEO analysis. DotBot adheres to the guidelines specified in a website's robots.txt file, ensuring that it only accesses pages that the site owner has permitted. This respectful approach helps maintain the integrity and performance of websites while allowing Moz to provide accurate and up-to-date SEO insights.
Identification
IP ranges
Requests from DotBot/1.2 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow DotBot/1.2
Add a Disallow rule for Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected]) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block DotBot/1.2 from crawling your entire site User-agent: Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected]) Disallow: / # Allow DotBot/1.2 full access User-agent: Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected]) Allow: /
Ensure your robots.txt allows Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected]). Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction