CriteoBot
GoodBotVerifiedBOTCriteo Crawler is a software that visits web pages and analyzes its content to serve relevant ads on them. Operated by Criteo.
What is CriteoBot?
CriteoBot is a web crawler operated by Criteo. Criteo Crawler is a software that visits web pages and analyzes its content to serve relevant ads on them.
Identification
CriteoBot/How we rated this bot
Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.
What we read
- google ignoring robots.txt ? - Google Search Central Community
- Where did Perplexity say they would obey robots.txt ... - Hacker News
- How to handle AI crawlers that ignore robots.txt - LinkedIn
- 20 Common Robots.txt Issues (and How to Avoid Them) - seoClarity
- Robots.txt Guide: Essential Rules & Disallow Best Practices
Why it crawls your site
CriteoBot crawls websites to collect data for advertising and marketing purposes. It is operated by Criteo as part of their advertising infrastructure. If you see this bot in your server logs, it is a verified crawler and generally safe.
Block or allow CriteoBot
Add a Disallow rule for CriteoBot/ in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block CriteoBot from crawling your entire site User-agent: CriteoBot/ Disallow: / # Allow CriteoBot full access User-agent: CriteoBot/ Allow: /
Ensure your robots.txt allows CriteoBot/. Verify requests are genuine by checking the user-agent string and referring to Criteo's documentation.
Traffic & trends
No Cloudflare traffic data available.
Loading trend data…
From the community
Recent Reddit discussion mentioning this bot.
Links & references
Data sources
This profile is compiled from the following sources.
Something wrong with this entry? Suggest a correction