Baiduspider/2.0

UndecidedCommunity-sourced

Baiduspider is the web crawler used by Baidu, China's leading search engine. The crawler is responsible for indexing content across the web to facilitate efficient search results for users. Baiduspider operates in a manner similar to other search engine crawlers, systematically browsing web pages to gather and update information in Baidu's extensive search database. Version 2.0 of Baiduspider includes enhancements that improve its efficiency and capabilities in crawling the web. It is designed to navigate various types of content across different languages and formats, ensuring comprehensive coverage of available information. The crawler respects the robots.txt protocol, allowing webmasters to control how their sites are indexed. Additionally, Baiduspider focuses on optimizing the crawling process to reduce server load and improve the speed at which new or updated content is reflected in search results. Overall, Baiduspider plays a crucial role in delivering relevant and timely search results to Baidu users, which is essential for maintaining the performance and reliability of the search engine.

Operator website ↗

Undecided

Assessment

Uncategorized

Identification

Bot name

Baiduspider/2.0

IP ranges

Requests from Baiduspider/2.0 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 207 ranges

Block or allow Baiduspider/2.0

Block it

Add a Disallow rule for Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Baiduspider/2.0 from crawling your entire site
User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Disallow: /

# Allow Baiduspider/2.0 full access
User-agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Allow: /

Allow & verify

Ensure your robots.txt allows Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html). Verify requests by checking the user-agent string.