CCBot/2.0

CCBot/2.0 is a web crawler developed by the Common Crawl Foundation, a non-profit organization dedicated to providing free access to web data for research and analysis. This crawler is based on the Apache Nutch project and utilizes the Apache Hadoop framework for processing and extracting crawl candidates. The user-agent string for CCBot/2.0 is: This string includes a link to the bot's information page, allowing website administrators to verify its identity. CCBot/2.0 is designed to minimize its impact on web servers. It employs an adaptive back-off algorithm that slows down requests if a server responds with HTTP 429 (Too Many Requests) or 5xx (Server Error) status codes. By default, the crawler waits a few seconds before sending the next request to the same site. To control the crawl rate, website administrators can specify a crawl delay in their robots.txt file. For example, to limit CCBot to one request every two seconds, the following lines can be added: This configuration instructs CCBot to wait at least two seconds between consecutive requests to the site. For more detailed information about CCBot/2.0, including its IP address ranges and compliance with the Robots Exclusion Protocol, please refer to the Common Crawl FAQ page.