How we track, verify, and rate every bot
Every entry in the registry follows the same process - from the moment a crawler first shows up in a data source to the sentiment rating you see on its page. Here is exactly how that works, and how to tell us when we get something wrong.
Overview
Bot WHOIS is an independent registry of automated web clients - AI crawlers, search engines, SEO tools, scrapers, monitors, and more. We aggregate signals from several independent sources, cross-reference them against operator-published records, and enrich each entry before it is listed. No operator pays to be included or to change its rating.
How we collect bots
A bot enters the registry from one of four kinds of source. Every candidate is de-duplicated by name and user-agent before it is enriched and listed.
Cloudflare Radar
We ingest Cloudflare Radar's verified-bot list and traffic data, which supplies the authoritative identity, category, and relative request-volume signal for a large share of the registry.
Crawler databases & operator docs
We pull from public crawler databases and the official user-agent and IP-range files operators publish, so a bot can be listed soon after its operator announces it.
Curated allow-lists
We cross-reference community-maintained crawler allow-lists (such as the widely-used GitHub lists) to catch well-known bots that our other sources miss.
Live discovery
Unknown user-agents seen in real request logs surface as candidates, which are researched and only published once they clear a confidence threshold.
Enrichment
Once a bot is identified, we enrich its profile from additional sources: a plain-language summary of what it does and why it crawls, its published IP ranges, block and allow guidance, recent community discussion, relevant news, and a search-interest trend. Enrichment runs continuously so profiles deepen over time.
Verification
A bot is marked Verified when an authoritative source - such as Cloudflare's verified-bot programme or an operator's own published records - confirms it is genuinely who it claims to be. Everything else is labelled Community-sourced: still listed, but not yet independently confirmed. A bot marked Botwhois.org is one we found ourselves: our analytics saw its traffic before any public list named it, and our research wrote it up.
Sentiment rating
Each bot gets one of three ratings, decided from public evidence and shown on its page with what we read. We follow the two tests in Cloudflare's verified-bot policy: does the bot identify itself honestly, and does it respect site owners (robots.txt and other opt-outs, reasonable request rates, no evading blocks)? We search for reports of it misbehaving and read what site owners say about it on Reddit. Training AI models is not bad in itself; ignoring an opt-out is. We do not watch bots on anyone's site, so confirm against your own logs before allowing or blocking.
Search results sometimes describe a different bot from the same company. When that happens an editor corrects the rating, and the bot's page says so and gives the reason beside the evidence we read.
Categorisation
Every bot is filed under one primary category based on what its traffic is for - training data, live assistance, search indexing, SEO tooling, scraping, monitoring, webhooks, and so on. Categories are kept coarse so the directory stays navigable; the full taxonomy with a live bot count for each lives on the Categories page.
Updates & corrections
The registry re-syncs on an automated schedule: sources are re-fetched, new bots are picked up, and existing profiles are re-enriched. Every bot page shows its last-updated date.
If you believe a rating or fact is wrong - especially if you operate the bot - tell us and we will review it against our sources and update the entry.