Directory/Categories/AI Crawler

AI Crawler

42 bots

Bots that gather public web pages to build training data for large language models.

Bots tracked

Flagged abusive

39 / 42

Verified

93%

Verified share

About AI Crawler bots

AI crawlers download publicly-accessible pages in bulk so their operators can build and refresh training datasets for large language models. The well-behaved ones announce themselves in the user-agent, publish their IP ranges, respect robots.txt, and provide a documented opt-out.

AI Crawler

Bots in this category

Every crawler and agent we classify as AI Crawler. Click through for identification, IP ranges, and block rules.

Bot · OperatorCategorySentimentTraffic · 14d

About AI Crawler bots

Bots in this category

Other categories