Directory/Categories/Search Engine Crawler/AIWebIndex-Agent

AIWebIndex-Agent

GoodBotVerifiedBOT

Fetches a page into Lyrenth's index when an agent requests one not yet indexed or wants a fresh copy. No training.

Operator website ↗

GoodBot

Assessment

Search Engine Crawler

Identification

Bot name

AIWebIndex-Agent

How we rated this bot

Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.

Names itself honestly

Yes

Verified by Cloudflare

Yes

Says it follows robots.txt and opt-outs

Not stated

Ignores site rules

None found

Disguises itself or evades blocks

None found

Overloads sites

None found

How widely people complain

No complaint about this bot in the evidence

Reddit discussion

There is no Reddit discussion of this bot in the evidence

What we read

Block or allow AIWebIndex-Agent

Block it

Add a Disallow rule for AIWebIndex-Agent in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block AIWebIndex-Agent from crawling your entire site
User-agent: AIWebIndex-Agent
Disallow: /

# Allow AIWebIndex-Agent full access
User-agent: AIWebIndex-Agent
Allow: /