ForestEngine
GoodBotBotwhois.orgForestEngine is an internet measurement scanner that probes public-facing services to collect TLS certificates, response headers, and service banners.
Identification
How we rated this bot
Identifies itself honestly, and we found no credible reports of abuse.
What we read
- I wonder if it will actually respect the robots.txt this time. | Hacker News
- google ignoring robots.txt ? - Google Search Central Community
- Several AI companies said to be ignoring robots dot txt ...
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- Does Robots.txt Matter Anymore? - Plagiarism Today
Block or allow ForestEngine
Add a Disallow rule for Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36 ForestEngine/1.0 (+https://forestengine.net/#opt-out) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block ForestEngine from crawling your entire site User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36 ForestEngine/1.0 (+https://forestengine.net/#opt-out) Disallow: / # Allow ForestEngine full access User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36 ForestEngine/1.0 (+https://forestengine.net/#opt-out) Allow: /
Ensure your robots.txt allows Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36 ForestEngine/1.0 (+https://forestengine.net/#opt-out). Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction