NE Crawler
UndecidedCommunity-sourcedThe netEstate NE Crawler is a web crawler developed by netEstate GmbH, designed to index and analyze web content for SEO purposes. It operates by continuously scanning the internet to gather data on various websites, which is then utilized to enhance search engine optimization strategies. ( netestate.de ) The crawler identifies itself with the user agent string: "netEstate NE Crawler (+ http://www.website-datenbank.de/ )," indicating its association with netEstate GmbH's website database services. ( user-agents.net ) In terms of technical specifications, the netEstate NE Crawler predominantly uses the IP address 81.209.177.145 and operates under the HTTP/1.1 protocol. It frequently requests file types such as TXT, JS, and CSS. ( robotsdb.de ) For more detailed information or inquiries, you can visit netEstate GmbH's official website at www.netestate.de . Operated by netEstate.
Identification
IP ranges
Requests from NE Crawler originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow NE Crawler
Add a Disallow rule for netEstate NE Crawler (+http://www.website-datenbank.de/) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block NE Crawler from crawling your entire site User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) Disallow: / # Allow NE Crawler full access User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) Allow: /
Ensure your robots.txt allows netEstate NE Crawler (+http://www.website-datenbank.de/). Verify requests by checking the user-agent string.
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction