Apify Website Content Crawler
GoodBotVerifiedAI AssistantAGENTOperated by Apify Docs
What is Apify Website Content Crawler?
Generated from Cloudflare, operator docs, and community data
Apify Website Content Crawler is a web crawler operated by Apify. Crawl websites and extract content to feed AI apps. Convert web data to Markdown or HTML, download files, and more.
Details
- Bot Name
- Apify Website Content Crawler
- Slug
- apify-website-content-crawler
- Kind
- AGENT
- Verification
- Verified by Cloudflare
- Category
- AI Assistant
- Operator
- Apify
User-Agent Patterns
ApifyWebsiteContentCrawlerFull User-Agent Strings
Main Use Cases
Why It Crawls Your Site
Apify Website Content Crawler crawls websites to collect and process web content. It is operated by Apify as part of their web crawling infrastructure. If you see this bot in your server logs, it is a verified crawler and generally safe.
How to Block / Whitelist
Block Apify Website Content Crawler
Add a Disallow rule for ApifyWebsiteContentCrawler in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.
# Block Apify Website Content Crawler from crawling your entire site User-agent: ApifyWebsiteContentCrawler Disallow: / # Allow Apify Website Content Crawler full access User-agent: ApifyWebsiteContentCrawler Allow: /
Whitelist Apify Website Content Crawler
Ensure your robots.txt allows ApifyWebsiteContentCrawler. Verify requests are genuine by checking the user-agent string and referring to Apify's documentation.
Traffic & Trends
Cloudflare Radar Traffic
No Cloudflare traffic data available
Google Trends Interest
Links & References
Data Sources
This profile was compiled from the following sources:
Last updated: May 26, 2026