Context7 Crawler

Context7 Crawler is an automated agent operated by Upstash, specifically engineered to index public-facing documentation. Unlike general-purpose AI crawlers that scrape data for large-scale model training, Context7 focuses on retrieving fresh, accurate content to enable AI systems to provide verifiable citations and links back to original source material. It operates with a transparent policy, identifying itself clearly in the user-agent string and providing robust support for standard web-governance protocols.

The bot crawls the web to build a searchable index of technical and public documentation, ensuring that AI-driven search tools have access to the most current information. By respecting robots.txt directives and 'noindex' tags, it allows site administrators to maintain control over their content's visibility. The operator provides a dedicated verification method using Web Bot Auth, allowing site owners to confirm the bot's identity and ensure it is not a malicious actor masquerading as a legitimate service.