DirectIndexBot
GoodBotVerifiedBOTScans business websites once to measure whether AI agents can find, read and act on them. Open index at directindex.org. Operated by DirectIndex.
What is DirectIndexBot?
DirectIndexBot is a diagnostic tool operated by DirectIndex that evaluates how business websites are rendered and interpreted by prominent AI agents, including GPTBot, ClaudeBot, and PerplexityBot. By simulating these specific crawlers and performing light browser-based checks, it generates reports that help site owners understand their AI-driven search presence. The bot is designed to be non-intrusive, strictly avoiding form submissions, logins, or data harvesting, and it provides clear mechanisms for site owners to opt-out via robots.txt.
Identification
How we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
What we read
- google ignoring robots.txt ? - Google Search Central Community
- I wonder if it will actually respect the robots.txt this time. | Hacker News
- How to deal with AI crawlers that ignore robots.txt | Soups Ranjan posted on the topic | LinkedIn
- Meta Has New Web-Crawling Bots That Sneakily Get Around Robots.txt - Business Insider
- 20 Common Robots.txt Issues (and How to Avoid Them)
Why it crawls your site
The bot crawls to perform a comparative analysis of how a website presents itself to different AI-driven search engines and chatbots. It fetches HTML content and specific metadata files to verify if a site is correctly configured for AI discovery or if it is being inadvertently blocked. The operator uses these findings to populate a public index of AI-friendly business sites, and the bot's identity can be verified through standard Web Bot Auth headers and Ed25519 signatures.
Block or allow DirectIndexBot
Add a Disallow rule for DirectIndexBot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block DirectIndexBot from crawling your entire site User-agent: DirectIndexBot Disallow: / # Allow DirectIndexBot full access User-agent: DirectIndexBot Allow: /
Ensure your robots.txt allows DirectIndexBot. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction