Meta-WebIndexer
GoodBotVerifiedBOTA web crawler operated by Meta designed to index online content to improve the relevance and accuracy of Meta AI search results. Operated by Meta.
What is Meta-WebIndexer?
Meta-WebIndexer is a specialized crawler deployed by Meta to gather information from across the web to power Meta AI. Unlike Meta's legacy social sharing crawlers, this agent focuses on content analysis to enhance the accuracy and relevance of AI-generated search responses. By indexing pages, it enables Meta AI to provide users with up-to-date information, citations, and direct links to the original source material.
Identification
meta-webindexer/1.1Why it crawls your site
The primary purpose of Meta-WebIndexer is to build a comprehensive knowledge base that allows Meta AI to understand and reference current web content. It crawls websites to extract text and metadata, which are then processed to improve the quality of AI interactions. Site owners can manage this crawler's access through standard robots.txt directives, ensuring that they maintain control over how their content is utilized within Meta's AI ecosystem.
Block or allow Meta-WebIndexer
Add a Disallow rule for meta-webindexer/1.1 in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Meta-WebIndexer from crawling your entire site User-agent: meta-webindexer/1.1 Disallow: / # Allow Meta-WebIndexer full access User-agent: meta-webindexer/1.1 Allow: /
Ensure your robots.txt allows meta-webindexer/1.1. Verify requests are genuine by checking the user-agent string and referring to Meta's documentation.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Related bots
Other crawlers operated by Meta.
Links & references
Data sources
This profile is compiled from the following sources.