Know exactly what's crawling your site.

An independent, regularly-updated reference for 1,181 web crawlers, AI bots, and spiders on the internet. See what people are saying about them, get the exact rules for blocking them, and find out who operates them.