Bot WHOIS
Submit a botLook up a bot
Directory/Categories/AI Crawler/AIWebIndex
AI

AIWebIndex

GoodBotBotwhois.org

AIWebIndex is a crawler that builds an AI-readable web index, transforming public web pages into structured 'AIDocuments' for AI agents to consume.

GoodBot
Assessment
AI Crawler
Category
Bot
Kind
1
Data sources

Identification

Bot name
AIWebIndex
Category
AI Crawler
Verification
Discovered and researched by Botwhois.org
User-agent string
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Main use cases
Building an AI-readable search indexProviding structured data for AI agentsContent indexing for LLM-based retrieval
Rated Sep 25, 2026 from public evidence

How we rated this bot

Identifies itself honestly, and we found no credible reports of abuse.

Names itself honestly
Yes
Verified by Cloudflare
No
Says it follows robots.txt and opt-outs
Not stated
Ignores site rules
None found
Disguises itself or evades blocks
None found
Overloads sites
None found
How widely people complain
No complaint about this bot in the evidence
Reddit discussion
There is no Reddit discussion of this bot in the evidence

What we read

Block or allow AIWebIndex

Block it

Add a Disallow rule for AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index) in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block AIWebIndex from crawling your entire site
User-agent: AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Disallow: /

# Allow AIWebIndex full access
User-agent: AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Allow: /
Allow & verify

Ensure your robots.txt allows AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index). Verify requests by checking the user-agent string.

Traffic & trends

Cloudflare Radar traffic

No Cloudflare traffic data available.

Google Trends interest

Loading trend data…

Links & references

Operator documentation
lyrenth.com
↗

Data sources

This profile is compiled from the following sources.

FathomDiscovery
Last updated September 25, 2026

Something wrong with this entry? Suggest a correction