SE
Search Engine Crawler
54 botsDiscovery-and-indexing crawlers that feed web, image, news, and video search results.
About Search Engine Crawler bots
Search-engine crawlers discover and index pages to build the results you see in web, image, news, and video search. They are the most established and best-behaved bots on the web, and blocking them usually means disappearing from that search engine entirely.
Search Engine Crawler
Bots in this category
Every crawler and agent we classify as Search Engine Crawler. Click through for identification, IP ranges, and block rules.
Bot · OperatorCategorySentimentTraffic · 14d
AL
Algolia
Algolia
Search Engine Crawler
GoodBot
—
›
AL
All Africa Crawler
AllAfrica Global Media
Search Engine Crawler
GoodBot
—
›
BA
Baidu ADS Server Proxy
Baidu
Search Engine Crawler
GoodBot
—
›
BA
Baiduspider
Baidu
Search Engine Crawler
GoodBot
+60%
›
BI
BingBot
Microsoft
Search Engine Crawler
GoodBot
+7%
›
CO
Coveo Bot
Coveo
Search Engine Crawler
GoodBot
—
›
CR
Crawlson
Crawlson
Search Engine Crawler
GoodBot
—
›
CC
Cốc Cốc
Coccoc
Search Engine Crawler
GoodBot
—
›
DA
Dataprovider.com
Dataprovider.com
Search Engine Crawler
GoodBot
—
›
DA
Daum
Daum
Search Engine Crawler
GoodBot
—
›
DU
DuckDuckBot
DuckDuckGo
Search Engine Crawler
GoodBot
—
›
FR
Freespoke
Freespoke
Search Engine Crawler
GoodBot
—
›
FU
Funnelback
Squiz - FunnelBack
Search Engine Crawler
GoodBot
—
›
GI
Gigabot
Gigablast
Search Engine Crawler
GoodBot
—
›
GO
Google Images
Google
Search Engine Crawler
GoodBot
—
›
GO
Google Scholar
Google
Search Engine Crawler
GoodBot
—
›
GO
Google Videos
Google
Search Engine Crawler
GoodBot
—
›
GO
Google-Flights-Search-Spackle
Google
Search Engine Crawler
GoodBot
—
›
GO
GoogleBot
Google
Search Engine Crawler
GoodBot
+9%
›
HY
Hype Machine
Hype Machine
Search Engine Crawler
GoodBot
—
›
IB
IbouBot
Babbar
Search Engine Crawler
GoodBot
—
›
KA
Kagi Bot
Kagi
Search Engine Crawler
GoodBot
—
›
LE
Level9SearchBot
Level 9
Search Engine Crawler
GoodBot
—
›
LI
Linespider
LINE Corporation
Search Engine Crawler
GoodBot
—
›
MR
MRGbot
MRG Web Services srl
Search Engine Crawler
GoodBot
—
›
MS
MSN
Microsoft
Search Engine Crawler
GoodBot
—
›
MA
MagiBot
Peak Labs
Search Engine Crawler
GoodBot
—
›
MA
Marginalia Search
Marginalia
Search Engine Crawler
GoodBot
—
›
MA
Mars Finder
Mars Flag
Search Engine Crawler
GoodBot
—
›
MO
Mojeek
Mojeek
Search Engine Crawler
GoodBot
—
›
NE
Neevabot
Neeva
Search Engine Crawler
GoodBot
—
›
NE
NewsNow
NewsNowUK
Search Engine Crawler
GoodBot
—
›
NO
Notabot
Helpfeel
Search Engine Crawler
GoodBot
—
›
PO
Potions
Potions
Search Engine Crawler
GoodBot
—
›
QW
Qwantbot
Qwant
Search Engine Crawler
GoodBot
—
›
SM
SMTnet PM Bot
SMTnet
Search Engine Crawler
GoodBot
—
›
SE
Seekport
SISTRIX
Search Engine Crawler
GoodBot
—
›
SE
Seznam
Seznam
Search Engine Crawler
GoodBot
—
›
SI
SiteSearch360
SEMKNOX GmbH
Search Engine Crawler
GoodBot
—
›
TE
Terracotta
Ceramic
Search Engine Crawler
GoodBot
—
›
TO
Toutiao
Bytedance
Search Engine Crawler
GoodBot
—
›
YA
Yahoo Japan SEO Crawler
LY Corporation
Search Engine Crawler
GoodBot
—
›
YA
Yahoo Slurp
Yahoo
Search Engine Crawler
GoodBot
—
›
YA
Yahoo! JAPAN
Yahoo! JAPAN (LY Corporation)
Search Engine Crawler
GoodBot
—
›
YA
YandexBot
Yandex
Search Engine Crawler
GoodBot
—
›
YE
Yeti
Naver
Search Engine Crawler
GoodBot
—
›
YO
YouBot
You.com
Search Engine Crawler
GoodBot
—
›
ZO
ZoomInfo
ZoomInfo
Search Engine Crawler
GoodBot
—
›
ZU
ZumBot
Zum Internet Corp.
Search Engine Crawler
GoodBot
—
›
BA
baidu
Baidu
Search Engine Crawler
GoodBot
—
›
DU
duckduckbot
DuckDuckGo
Search Engine Crawler
GoodBot
—
›
GO
google-xrawler
Google
Search Engine Crawler
GoodBot
—
›
JO
jobswithgptcom-bot
JobsWithGPT
Search Engine Crawler
GoodBot
—
›
YA
yandexbot
Yandex
Search Engine Crawler
GoodBot
—
›
Other categories
Browse the rest of the bot taxonomy.