WebSpiderMount
GoodBotVerifiedBOTA specialized web scraping and job wrapping service that extracts recruitment data to populate job boards and employment marketplaces. Operated by Aspen Tech Labs.
What is WebSpiderMount?
WebSpiderMount is a professional data extraction service operated by Aspen Tech Labs, specifically designed for the recruitment and employment industry. It functions as a 'job wrapper,' systematically crawling employer career pages and job boards to normalize and syndicate job listings for major platforms like ZipRecruiter, CareerBuilder, and Dice. By automating the collection of job metadata, it enables recruitment sites to maintain up-to-date listings without manual entry.
Identification
webspidermountWhy it crawls your site
The bot crawls websites to extract structured job data, such as titles, descriptions, locations, and application links, which are then integrated into the client's job board ecosystem. It operates to ensure that job postings remain synchronized across various platforms in real-time. Site owners can verify the bot's activity by checking the user-agent string against the official documentation at webspidermount.com and can manage its access via standard robots.txt directives.
Block or allow WebSpiderMount
Add a Disallow rule for webspidermount in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block WebSpiderMount from crawling your entire site User-agent: webspidermount Disallow: / # Allow WebSpiderMount full access User-agent: webspidermount Allow: /
Ensure your robots.txt allows webspidermount. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.