Webindexer/1.1

UndecidedCommunity-sourced

Webindexer/1.1 is a crawler developed by Facebook that is used for indexing web pages and gathering information for the platform's sharing features. This crawler helps ensure that when users share links on Facebook, the content that appears in the previews (such as titles, descriptions, and images) accurately reflects the linked page. The crawler follows specific protocols set forth by webmasters, including the use of the robots.txt file to respect the rules and guidelines for indexing. By adhering to these protocols, Webindexer/1.1 helps maintain proper web etiquette while efficiently collecting data necessary for enhancing user experience on Facebook. Webindexer contributes to the site's ability to generate rich previews, which are crucial for engaging users and driving traffic. It is also designed to handle dynamic content and may revisit pages to ensure that any updates are captured for accurate sharing. For more detailed information about how Webindexer interacts with web pages, refer to the official Facebook documentation on web crawlers. Operated by Meta.

Operator website ↗

Undecided

Assessment

Uncategorized

Identification

Bot name

Webindexer/1.1

Operator

IP ranges

Requests from Webindexer/1.1 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 369 ranges

Block or allow Webindexer/1.1

Block it

Add a Disallow rule for Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Webindexer/1.1 from crawling your entire site
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Disallow: /

# Allow Webindexer/1.1 full access
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Allow: /

Allow & verify

Ensure your robots.txt allows Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)). Verify requests by checking the user-agent string.