Meta-ExternalAgent/1.1

FacebookExternalHit is a web crawler used by Meta to fetch and display content from websites and apps shared on platforms like Facebook, Instagram, and Messenger. When a link is shared, this crawler retrieves information such as the title, description, and thumbnail image to present a rich preview. The User-Agent strings identifying this crawler include: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) facebookexternalhit/1.1 facebookcatalog/1.0 It's important to note that FacebookExternalHit may bypass robots.txt directives during security or integrity checks, such as scanning for malware or malicious content. ( developers.facebook.com ) To ensure optimal interaction with this crawler, consider the following recommendations: Server Configuration : Ensure your server supports gzip and deflate encodings. Open Graph Properties : Place any Open Graph properties within the first 1 MB of your website's content to prevent truncation. Crawl Efficiency : Design your website to allow the crawler to access content within a few seconds to facilitate proper display. Response Handling : Your website should generate and return responses with all required properties, adhering to the bytes specified in the Range header of the crawler's request, or it should ignore the Range header altogether. Allow List : Add the crawler's User-Agent strings or IP addresses to your allow list for enhanced security. If your website's content isn't available during the crawler's visit, you can prompt a re-crawl by submitting the URL through the Sharing Debugger tool or by using the Sharing API. ( developers.facebook.com ) For more detailed information, refer to Meta's documentation on web crawlers. ( developers.facebook.com )