Directory/Categories/SEO/Cloudflare Prefetch

Cloudflare Prefetch

Name: Cloudflare Prefetch
Author: Cloudflare

GoodBotVerifiedBOT

URL prefetching means that Cloudflare pre-populates the cache with content a visitor is likely to request next. This setting leads to a higher cache hit rate and thus a faster experience for the user. (https://developers.cloudflare.com/fundamentals/speed/prefetch-urls/) Operated by Cloudflare.

Operator website ↗

GoodBot

Assessment

SEO

What is Cloudflare Prefetch?

Cloudflare Prefetch is a web crawler operated by Cloudflare. URL prefetching means that Cloudflare pre-populates the cache with content a visitor is likely to request next. This setting leads to a higher cache hit rate and thus a faster experience for the user. (https://developers.cloudflare.com/fundamentals/speed/prefetch-urls/)

Helpful — Verified, safe crawler. Respects robots.txt and provides operator documentation.

Identification

Bot name

Cloudflare Prefetch

Operator

Cloudflare

Why it crawls your site

Cloudflare Prefetch crawls websites to analyze search engine optimization factors and backlinks. It is operated by Cloudflare as part of their SEO analysis infrastructure. If you see this bot in your server logs, it is a verified crawler and generally safe.

Block or allow Cloudflare Prefetch

Block it

Add a Disallow rule for CloudFlare-Prefetch in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block Cloudflare Prefetch from crawling your entire site
User-agent: CloudFlare-Prefetch
Disallow: /

# Allow Cloudflare Prefetch full access
User-agent: CloudFlare-Prefetch
Allow: /

Allow & verify

Ensure your robots.txt allows CloudFlare-Prefetch. Verify requests are genuine by checking the user-agent string and referring to Cloudflare's documentation.