VelenPublicWebCrawler/1.0
GoodBotCommunity-sourcedVelenPublicWebCrawler/1.0 is a web crawler developed by Velen, a company specializing in data analysis and machine learning. This crawler is designed to analyze millions of publicly accessible internet pages each month, focusing solely on content that is not behind authentication barriers. Its primary objective is to build business datasets and machine learning models to enhance the understanding of the web. ( velen.io ) The crawler adheres to the directives specified in robots.txt files and meta tags, ensuring compliance with website owners' preferences. To minimize the impact on website performance, VelenPublicWebCrawler typically accesses one page at a time, with a delay of approximately two seconds between requests. ( velen.io ) If website administrators wish to block this crawler, they can do so by adding the following directive to their robots.txt file: This instruction will prevent the crawler from accessing the site's pages. ( velen.io ) For more information or to provide feedback, you can contact Velen at [email protected]. ( velen.io ) Operated by Velen.
What is VelenPublicWebCrawler/1.0?
VelenPublicWebCrawler is an automated agent developed by Velen, the company behind the email discovery platform Hunter.io. It systematically traverses the public web to aggregate data for the purpose of constructing comprehensive business datasets and training machine learning models. The bot is designed with a focus on ethical crawling, adhering to standard web protocols such as robots.txt and meta tags to ensure it only accesses publicly available information.
Identification
How we rated this bot
Identifies itself honestly, and we found no credible reports of abuse.
What we read
Why it crawls your site
The crawler operates to support the data infrastructure of Hunter.io, enabling the platform to maintain accurate and up-to-date business contact information. By analyzing millions of pages, it extracts structured data that powers their search tools. The operator provides a clear contact channel for feedback and explicitly supports opt-out mechanisms via robots.txt, allowing site owners to easily restrict its access if desired.
IP ranges
Requests from VelenPublicWebCrawler/1.0 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow VelenPublicWebCrawler/1.0
Add a Disallow rule for Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block VelenPublicWebCrawler/1.0 from crawling your entire site User-agent: Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) Disallow: / # Allow VelenPublicWebCrawler/1.0 full access User-agent: Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) Allow: /
Ensure your robots.txt allows Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io). Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction