Bot WHOIS
Submit a botLook up a bot
DirectoryOperatorsCategoriesMethodologyAPIAboutContact
Submit a botLook up a bot
Directory/Categories/Other/Jscrawler/0.1
JS

Jscrawler/0.1

UndecidedCommunity-sourced

Jscrawler is a lightweight, open-source utility designed to fetch and extract JavaScript files from specified URLs or subdomains.

Undecided
Assessment
Other
Category
Bot
Kind
1
Data sources
AI-generated summary, drawn from Cloudflare, operator docs, and community data

What is Jscrawler/0.1?

Jscrawler is an open-source tool, commonly found on platforms like GitHub, used primarily by security researchers and developers to automate the discovery and retrieval of JavaScript files from target domains. Unlike enterprise-grade crawlers, it is a simple, high-performance script that does not inherently respect standard web crawling protocols like robots.txt. It is frequently utilized in bug bounty programs and security audits to identify client-side vulnerabilities or hidden API endpoints within web applications.

Unknown — While the tool itself is a standard utility for security researchers and developers, its lack of adherence to robots.txt and its potential for mass-scraping make it a tool that can be easily misused for unauthorized data collection.

Identification

Bot name
Jscrawler/0.1
Category
Other
Verification
Community-sourced
User-agent string
Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)
Main use cases
JavaScript file discoverySecurity reconnaissanceWeb asset enumeration

Why it crawls your site

The bot crawls to perform automated reconnaissance on web infrastructure. By fetching JavaScript files, it allows operators to analyze source code for sensitive information, hardcoded credentials, or insecure logic that might be exploitable. Because it is an open-source project, it is often deployed by individual users rather than a centralized organization, making its behavior highly dependent on the intent of the person running the script.

IP ranges

Requests from Jscrawler/0.1 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Block or allow Jscrawler/0.1

Block it

Add a Disallow rule for Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Jscrawler/0.1 from crawling your entire site
User-agent: Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)
Disallow: /

# Allow Jscrawler/0.1 full access
User-agent: Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)
Allow: /
Allow & verify

Ensure your robots.txt allows Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/). Verify requests by checking the user-agent string.

Traffic & trends

Cloudflare Radar traffic

No Cloudflare traffic data available.

Google Trends interest

Loading trend data…

Links & references

Operator documentation
github.com
↗

Data sources

This profile is compiled from the following sources.

CrawlersDatabase
Last updated September 18, 2026

Something wrong with this entry? Suggest a correction