SemrushBot/7~bl

GoodBotCommunity-sourced

SemrushBot/7~bl is a web crawler developed by Semrush Inc., designed to discover and collect new and updated web data. This data is utilized across various Semrush tools, including Backlink Analytics, Site Audit, Backlink Audit, Link Building, SEO Writing Assistant, On Page SEO Checker, SEO Content Template, Topic Research, SplitSignal, ContentShake AI, and Plagiarism Checker. ( semrush.com ) The user agent string for SemrushBot/7~bl is: SemrushBot operates by starting with a list of webpage URLs. As it visits these URLs, it saves hyperlinks from the page for further crawling. This process, known as the "crawl frontier," is repeatedly visited according to Semrush's policies to effectively map a site for updates, such as content changes, new pages, and dead links. ( semrush.com ) If you prefer to prevent SemrushBot from crawling your site, you can do so by adding the following lines to your robots.txt file: This directive instructs SemrushBot to refrain from accessing any pages on your website. ( semrush.com ) Please note that while SemrushBot is designed to adhere to robots.txt directives, some users have reported challenges in blocking the bot using this method. In such cases, alternative measures, such as blocking the bot by IP address or user agent at the server level, may be more effective. ( stackoverflow.com ) Operated by Semrush.

Operator website ↗

GoodBot

Assessment

Uncategorized

Identification

Bot name

SemrushBot/7~bl

Operator

Semrush

How we rated this bot

This is Semrush's main crawler. It names itself honestly and Semrush documents how to block it with robots.txt. Most of what we found is site owners choosing to block SEO crawlers, not evidence that it ignores their rules.

Names itself honestly

Yes

Verified by Cloudflare

Says it follows robots.txt and opt-outs

Not stated

Ignores site rules

Credible reports found

Disguises itself or evades blocks

None found

Overloads sites

None found

How widely people complain

Several independent site owners report the same problem

Reddit discussion

There is no Reddit discussion of this bot in the evidence

What we read

IP ranges

Requests from SemrushBot/7~bl originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 45 ranges

Block or allow SemrushBot/7~bl

Block it

Add a Disallow rule for Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block SemrushBot/7~bl from crawling your entire site
User-agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Disallow: /

# Allow SemrushBot/7~bl full access
User-agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Allow: /

Allow & verify

Ensure your robots.txt allows Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html). Verify requests by checking the user-agent string.