Webindexer/1.1
GoodBotCommunity-sourcedWebindexer/1.1 is a crawler developed by Facebook that is used for indexing web pages and gathering information for the platform's sharing features. This crawler helps ensure that when users share links on Facebook, the content that appears in the previews (such as titles, descriptions, and images) accurately reflects the linked page. The crawler follows specific protocols set forth by webmasters, including the use of the robots.txt file to respect the rules and guidelines for indexing. By adhering to these protocols, Webindexer/1.1 helps maintain proper web etiquette while efficiently collecting data necessary for enhancing user experience on Facebook. Webindexer contributes to the site's ability to generate rich previews, which are crucial for engaging users and driving traffic. It is also designed to handle dynamic content and may revisit pages to ensure that any updates are captured for accurate sharing. For more detailed information about how Webindexer interacts with web pages, refer to the official Facebook documentation on web crawlers. Operated by Meta.
What is Webindexer/1.1?
Meta-WebIndexer is a specialized crawler developed by Meta to index public web pages for the purpose of enhancing the search capabilities of Meta AI. Unlike Meta's other crawlers, such as those used for social media link previews or ad-related data, this agent is specifically tasked with gathering information to ensure that Meta AI can provide accurate, cited, and relevant responses to user queries. It operates under the 'meta-webindexer' token and respects standard robots.txt directives, allowing site administrators to manage how their content is utilized within Meta's AI ecosystem.
Identification
How we rated this bot
Identifies itself honestly, and we found no credible reports of abuse.
What we read
Why it crawls your site
The primary purpose of Meta-WebIndexer is to build and maintain a comprehensive index of the web that supports Meta AI's search functionality. By crawling and analyzing public content, the bot enables Meta AI to reference and link back to original sources, providing users with verifiable information. Meta encourages site owners to allow this crawler to ensure their content remains eligible for inclusion and citation in AI-generated search results, and the bot's activity can be verified through its documented User-Agent and adherence to the robots.txt protocol.
IP ranges
Requests from Webindexer/1.1 originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow Webindexer/1.1
Add a Disallow rule for Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Webindexer/1.1 from crawling your entire site User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) Disallow: / # Allow Webindexer/1.1 full access User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) Allow: /
Ensure your robots.txt allows Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)). Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Related bots
Other crawlers operated by Meta.
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction