googlebot
GoodBotCommunity-sourcedThe primary web crawler used by Google to discover, index, and render web pages for inclusion in Google Search results. Operated by Google.
What is googlebot?
Googlebot is the foundational crawler operated by Google that traverses the internet to build the index for Google Search. It operates as two distinct types: a desktop crawler and a mobile-first crawler, both of which render JavaScript to ensure modern, dynamic web applications are indexed accurately. By adhering to robots.txt directives and crawl budget management, it balances the need for fresh content with the performance requirements of the websites it visits.
Identification
Why it crawls your site
Googlebot crawls the web to populate Google's search index, allowing the search engine to provide relevant results to user queries. It processes HTML, CSS, and JavaScript to understand the structure and content of pages, which informs ranking algorithms. Site owners can verify the authenticity of this bot by performing a reverse DNS lookup on the requesting IP address to ensure it resolves to a googlebot.com domain, or by using the official Google Search Console tools.
IP ranges
Requests from googlebot originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.
Block or allow googlebot
Add a Disallow rule for googlebot in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block googlebot from crawling your entire site User-agent: googlebot Disallow: / # Allow googlebot full access User-agent: googlebot Allow: /
Ensure your robots.txt allows googlebot. Verify requests by checking the user-agent string.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Related bots
Other crawlers operated by Google.
Data sources
This profile is compiled from the following sources.