Directory/Categories/Meta-webindexer/1.1

Meta-webindexer/1.1

UndecidedCommunity-sourced

Meta-webindexer/1.1 is a web crawler used by Facebook to index content from various websites for its sharing features, such as the Open Graph protocol. This crawler visits web pages to collect information and metadata that can enhance how content is displayed when shared on the Facebook platform. According to the documentation, Meta-webindexer functions by fetching metadata available in the HTML of web pages, which can include titles, descriptions, images, and other Open Graph tags. The crawler helps ensure that shared links are rendered correctly and provide a rich media experience for users. It adheres to the rules specified in the robots.txt file and respects crawl delay settings set by webmasters. Understanding how the Meta-webindexer operates can help webmasters optimize their websites for better visibility on Facebook by correctly implementing Open Graph tags and complying with crawling guidelines. Operated by Meta.

Operator website ↗

Undecided

Assessment

Uncategorized

Identification

Bot name

Meta-webindexer/1.1

Operator

IP ranges

Requests from Meta-webindexer/1.1 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 396 ranges

Block or allow Meta-webindexer/1.1

Block it

Add a Disallow rule for Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Meta-webindexer/1.1 from crawling your entire site
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Disallow: /

# Allow Meta-webindexer/1.1 full access
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Allow: /

Allow & verify

Ensure your robots.txt allows Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)). Verify requests by checking the user-agent string.