Directory/Categories/Verified Bot/google-special-crawlers

google-special-crawlers

Name: google-special-crawlers
Author: Google

GoodBotCommunity-sourced

A specialized crawler used by Google for specific, non-standard indexing tasks that fall outside the scope of the primary Googlebot. Operated by Google.

GoodBot

Assessment

Verified Bot

What is google-special-crawlers?

The google-special-crawlers agent is a dedicated crawler operated by Google to handle unique indexing requirements that differ from the standard web-crawling behavior of Googlebot. It is typically deployed for specific Google services or features that require targeted data extraction rather than general-purpose web discovery. By isolating these tasks, Google maintains better control over resource allocation and ensures that specialized content is processed according to the requirements of specific Google products.

Helpful — This is a verified Google agent that performs specialized indexing tasks to ensure content is correctly represented in Google's ecosystem.

Identification

Bot name

google-special-crawlers

Operator

Google

Why it crawls your site

This bot crawls to fulfill specific data requirements for Google's internal services, such as specialized search features, product integrations, or content verification processes. It operates under the same infrastructure as other Google crawlers, respecting standard robots.txt directives and site-level crawl configurations. Site owners can verify this bot by performing a reverse DNS lookup on the originating IP address to confirm it resolves to a googlebot.com domain.

IP ranges

Requests from google-special-crawlers originate from these published ranges. For strict verification, match the source IP against this list in addition to checking the user-agent and reverse-DNS.

Showing 24 of 577 ranges

Block or allow google-special-crawlers

Block it

Add a Disallow rule for google-special-crawlers in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block google-special-crawlers from crawling your entire site
User-agent: google-special-crawlers
Disallow: /

# Allow google-special-crawlers full access
User-agent: google-special-crawlers
Allow: /