SE
Search Engine Crawler
120 botsDiscovery-and-indexing crawlers that feed web, image, news, and video search results.
About Search Engine Crawler bots
Search-engine crawlers discover and index pages to build the results you see in web, image, news, and video search. They are the most established and best-behaved bots on the web, and blocking them usually means disappearing from that search engine entirely.
Search Engine Crawler
Bots in this category
Every crawler and agent we classify as Search Engine Crawler. Click through for identification, IP ranges, and block rules.
Bot · OperatorCategorySentimentTraffic · 14d
AIWebIndex
Unknown operator
Search Engine Crawler
GoodBot
—
›
AI
AIWebIndex-Agent
Unknown operator
Search Engine Crawler
GoodBot
—
›
Algolia
Algolia
Search Engine Crawler
GoodBot
—
›
All Africa Crawler
AllAfrica Global Media
Search Engine Crawler
GoodBot
—
›
AP
Apply Anyway
Unknown operator
Search Engine Crawler
GoodBot
—
›
Baidu ADS Server Proxy
Baidu
Search Engine Crawler
GoodBot
—
›
Baiduspider
Baidu
Search Engine Crawler
GoodBot
+11%
›
BR
Brief
Unknown operator
Search Engine Crawler
GoodBot
—
›
CA
Capsulae
Unknown operator
Search Engine Crawler
GoodBot
—
›
CE
Centry AdvisorAI
Unknown operator
Search Engine Crawler
GoodBot
—
›
Channel3Bot
Channel3
Search Engine Crawler
GoodBot
—
›
Cloudflare Web Scanner
Cloudflare, Inc.
Search Engine Crawler
GoodBot
—
›
Context7 Crawler
Upstash
Search Engine Crawler
GoodBot
—
›
Coveo Bot
Coveo
Search Engine Crawler
GoodBot
—
›
Crawlson
Crawlson
Search Engine Crawler
GoodBot
—
›
CuteStat
CuteStat
Search Engine Crawler
GoodBot
—
›
Cốc Cốc
Coccoc
Search Engine Crawler
GoodBot
—
›
Dataprovider.com
Dataprovider.com
Search Engine Crawler
GoodBot
—
›
Daum
Daum
Search Engine Crawler
GoodBot
—
›
EL
EltropyBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
EN
EnrolleBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
EV
EveryIowan-Hydra
Unknown operator
Search Engine Crawler
GoodBot
—
›
ExaSearchBot
Exa
Search Engine Crawler
GoodBot
—
›
FI
FindAGovJobBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Freespoke
Freespoke
Search Engine Crawler
GoodBot
—
›
Funnelback
Squiz - FunnelBack
Search Engine Crawler
GoodBot
—
›
GE
GEFTBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Gigabot
Gigablast
Search Engine Crawler
GoodBot
—
›
Google Images
Google
Search Engine Crawler
GoodBot
—
›
Google Scholar
Google
Search Engine Crawler
GoodBot
—
›
Google Videos
Google
Search Engine Crawler
GoodBot
—
›
Google-Flights-Search-Spackle
Google
Search Engine Crawler
GoodBot
—
›
HA
HamletBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
HO
HomininBot
Hominin
Search Engine Crawler
GoodBot
—
›
Hype Machine
Hype Machine
Search Engine Crawler
GoodBot
—
›
IbouBot
Babbar
Search Engine Crawler
GoodBot
—
›
InstantAIGuruBot
InstantAIGuru
Search Engine Crawler
GoodBot
—
›
IS
IsonSearchBot
Epithre
Search Engine Crawler
GoodBot
—
›
JobicyBot
Jobicy
Search Engine Crawler
GoodBot
—
›
Kagi Bot
Kagi
Search Engine Crawler
GoodBot
—
›
KA
KalicartGlobalBot
KaliCart Global
Search Engine Crawler
GoodBot
—
›
KeenableBot
Keenable.ai, Inc.
Search Engine Crawler
GoodBot
—
›
KeldanNewsCrawler
Keldan
Search Engine Crawler
GoodBot
—
›
KO
Kongi
Unknown operator
Search Engine Crawler
GoodBot
—
›
KR
KrasaBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Level9SearchBot
Level 9
Search Engine Crawler
GoodBot
—
›
Linespider
LINE Corporation
Search Engine Crawler
GoodBot
—
›
LinkupBot
Linkup
Search Engine Crawler
GoodBot
—
›
MRGbot
MRG Web Services srl
Search Engine Crawler
GoodBot
—
›
MSN
Microsoft
Search Engine Crawler
GoodBot
—
›
MS
MSNBot
Microsoft
Search Engine Crawler
GoodBot
—
›
MagiBot
Peak Labs
Search Engine Crawler
GoodBot
—
›
Marginalia Search
Marginalia
Search Engine Crawler
GoodBot
—
›
Mars Finder
Mars Flag
Search Engine Crawler
GoodBot
—
›
MistralAI-Index
Mistral AI
Search Engine Crawler
GoodBot
—
›
Mojeek
Mojeek
Search Engine Crawler
GoodBot
—
›
MY
My Local Obits
Unknown operator
Search Engine Crawler
GoodBot
—
›
MY
MyPortaBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Neevabot
Neeva
Search Engine Crawler
GoodBot
—
›
NE
Nekuda WebMCP Discovery Bot
Nekuda
Search Engine Crawler
GoodBot
—
›
NewsNow
NewsNowUK
Search Engine Crawler
GoodBot
—
›
Notabot
Helpfeel
Search Engine Crawler
GoodBot
—
›
ON
ONE Leilões
Unknown operator
Search Engine Crawler
GoodBot
—
›
OC
OctenSearchBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
PI
PinLabrador
Unknown operator
Search Engine Crawler
GoodBot
—
›
Platebreaker
Unknown operator
Search Engine Crawler
GoodBot
—
›
Potions
Potions
Search Engine Crawler
GoodBot
—
›
QU
QuntiBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Qwantbot
Qwant
Search Engine Crawler
GoodBot
—
›
QW
Qwantbot/1.0
Qwant
Search Engine Crawler
Undecided
—
›
RDTvlokipBot
Théo CHARLET
Search Engine Crawler
GoodBot
—
›
SMTnet PM Bot
SMTnet
Search Engine Crawler
GoodBot
—
›
Seekport
SISTRIX
Search Engine Crawler
GoodBot
—
›
SE
SeekportBot
SISTRIX
Search Engine Crawler
GoodBot
—
›
Seznam
Seznam
Search Engine Crawler
GoodBot
—
›
SE
SeznamBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
SE
SeznamBot/4.0
Seznam.cz
Search Engine Crawler
GoodBot
—
›
SH
ShotGeniusBot
FJB SAS
Search Engine Crawler
GoodBot
—
›
SiteSearch360
SEMKNOX GmbH
Search Engine Crawler
GoodBot
—
›
SO
SofyaBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
SO
SojournerBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
SO
SoothBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
SP
SpacepatrolSearch
Unknown operator
Search Engine Crawler
GoodBot
—
›
SU
SuggestAPI-CatalogBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Terracotta
Ceramic
Search Engine Crawler
GoodBot
—
›
Toutiao
Bytedance
Search Engine Crawler
GoodBot
—
›
TU
TulvezBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
UP
UpContent
UpContent
Search Engine Crawler
GoodBot
—
›
UP
UpContent QA
Unknown operator
Search Engine Crawler
GoodBot
—
›
VI
ViaCatholica
Unknown operator
Search Engine Crawler
GoodBot
—
›
WE
WebObservationLab
Unknown operator
Search Engine Crawler
GoodBot
—
›
WE
Webmaster.petalsearch
Huawei Technologies Co., Ltd.
Search Engine Crawler
GoodBot
—
›
WI
WindowsForum-AI
WindowsForum.com
Search Engine Crawler
GoodBot
—
›
WR
WrenBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
Yahoo Slurp
Yahoo
Search Engine Crawler
GoodBot
—
›
Yahoo! JAPAN
Yahoo! JAPAN (LY Corporation)
Search Engine Crawler
GoodBot
—
›
Yeti
Naver
Search Engine Crawler
GoodBot
—
›
YI
YigoBot
KVCHub
Search Engine Crawler
GoodBot
—
›
YouBot
You.com
Search Engine Crawler
GoodBot
—
›
ZoomInfo
ZoomInfo
Search Engine Crawler
GoodBot
—
›
ZumBot
Zum Internet Corp.
Search Engine Crawler
GoodBot
—
›
AI
aidealioBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
baidu
Baidu
Search Engine Crawler
Undecided
—
›
bingbot
Microsoft
Search Engine Crawler
GoodBot
+21%
›
CE
cercadorbot
Unknown operator
Search Engine Crawler
GoodBot
—
›
duckduckbot
DuckDuckGo
Search Engine Crawler
GoodBot
—
›
ET
etrite bot
Etrite
Search Engine Crawler
GoodBot
—
›
google-xrawler
Google
Search Engine Crawler
GoodBot
—
›
googlebot
Google
Search Engine Crawler
GoodBot
−12%
›
HA
hankbot
hank.parts S. L.
Search Engine Crawler
GoodBot
—
›
HE
healthcareersbot
Cleared Systems, LLC
Search Engine Crawler
GoodBot
—
›
i-search-crawler
Scala Communications, Inc.
Search Engine Crawler
GoodBot
—
›
IB
ibou-search
Unknown operator
Search Engine Crawler
GoodBot
—
›
jobswithgptcom-bot
JobsWithGPT
Search Engine Crawler
GoodBot
—
›
oai-searchbot
OpenAI
Search Engine Crawler
GoodBot
—
›
OD
odosano-bot
Unknown operator
Search Engine Crawler
GoodBot
—
›
PF
pfindBot
Unknown operator
Search Engine Crawler
GoodBot
—
›
SO
sourcedash
Source Dashboard
Search Engine Crawler
GoodBot
—
›
SY
synffbot
Unknown operator
Search Engine Crawler
GoodBot
—
›
yandexbot
Yandex
Search Engine Crawler
GoodBot
—
›
Other categories
Browse the rest of the bot taxonomy.