semrush
GoodBotCommunity-sourcedA specialized crawler used by the Semrush platform to gather data for search engine optimization, backlink analysis, and competitive intelligence. Operated by Semrush.
What is semrush?
SemrushBot is the primary automated agent operated by Semrush, a leading digital marketing suite. It traverses the web to index pages, analyze link structures, and track keyword rankings, which are then processed to provide users with actionable SEO and marketing intelligence. The bot is designed to respect robots.txt directives and is a standard tool for webmasters looking to monitor their site's performance against industry benchmarks.
Identification
Why it crawls your site
The bot crawls the web to build and maintain a massive database of backlinks, organic search rankings, and domain authority metrics. This data allows Semrush users to perform site audits, identify technical SEO issues, and track how their content performs in search engine results pages. Site owners can verify the bot by checking that the reverse DNS lookup of the requesting IP address resolves to the semrush.com domain.
IP ranges
Requests from semrush originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow semrush
Add a Disallow rule for semrush in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block semrush from crawling your entire site User-agent: semrush Disallow: / # Allow semrush full access User-agent: semrush Allow: /
Ensure your robots.txt allows semrush. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Related bots
Other crawlers operated by Semrush.
Data sources
This profile is compiled from the following sources.