Reflectionbot
GoodBotVerifiedBOTReflectionbot is the official web crawler for Reflection AI, designed to index publicly available web content to support the development of open-source AI models. Operated by Reflection AI.
What is Reflectionbot?
Reflectionbot is operated by Reflection AI to gather data for the creation of open-source artificial intelligence models. Unlike many proprietary crawlers, the project emphasizes transparency by providing a dedicated documentation portal, clear instructions for robots.txt configuration, and verifiable IP ranges. The bot is designed to be identifiable through a specific user-agent token and reverse DNS lookups, allowing site owners to manage its access effectively.
Identification
ReflectionbotHow we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
What we read
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- 20 Common Robots.txt Issues (and How to Avoid Them)
- robots.txt - Wikipedia
- Do you respect robots.txt? | Hacker News
- Problems with web crawlers not respecting robots.txt file - Application Security / SSL / TLS - Cloudflare Community
Why it crawls your site
The primary purpose of Reflectionbot is to ingest public web data to train and refine Reflection AI's machine learning models. By crawling the web, the bot builds the datasets necessary for the company's mission of developing open-source intelligence tools. Site owners can verify the bot's legitimacy by checking that the originating IP addresses resolve to the reflection.ai domain, ensuring that the traffic is indeed from the official operator.
Block or allow Reflectionbot
Add a Disallow rule for Reflectionbot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Reflectionbot from crawling your entire site User-agent: Reflectionbot Disallow: / # Allow Reflectionbot full access User-agent: Reflectionbot Allow: /
Ensure your robots.txt allows Reflectionbot. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction