AdsTxtCrawler

GoodBotVerifiedAdvertising & MarketingBOT

Operated by OneTag Limited Docs

What is AdsTxtCrawler?

Generated from Cloudflare, operator docs, and community data

The AdsTxtCrawler, operated by OneTag Limited, is designed to discover and parse ads.txt files on websites. These files are part of the IAB Tech Lab's Authorized Digital Sellers (ads.txt) initiative, which aims to combat ad fraud by allowing publishers to declare authorized sellers of their digital advertising inventory. The crawler identifies these files, extracts the data, and likely uses it to build a comprehensive view of the digital advertising ecosystem, helping to ensure transparency and prevent unauthorized reselling of ad space.

Helpful — Aids in verifying legitimate advertising sources.

Details

Bot Name
AdsTxtCrawler
Slug
adstxtcrawler
Kind
BOT
Verification
Verified by Cloudflare
Category
Advertising & Marketing
Operator
OneTag Limited

User-Agent Patterns

InteractiveAdvertisingBureau

Full User-Agent Strings

AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler

Main Use Cases

Crawling for ads.txt filesAnalyzing advertising authorizationsVerifying legitimate ad sellers

Why It Crawls Your Site

AdsTxtCrawler crawls the web to locate ads.txt files, which contain a list of authorized sellers for a given website's ad inventory. By collecting and analyzing this data, OneTag Limited can help advertisers and publishers verify the legitimacy of ad sellers and prevent ad fraud. The data gathered by the crawler is used to maintain an up-to-date registry of authorized digital sellers, contributing to a more transparent and secure advertising ecosystem. The bot identifies itself via its user agent and links to the IAB's GitHub repository for the project, allowing easy verification.

How to Block / Whitelist

Block AdsTxtCrawler

Add a Disallow rule for InteractiveAdvertisingBureau in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block AdsTxtCrawler from crawling your entire site
User-agent: InteractiveAdvertisingBureau
Disallow: /

# Allow AdsTxtCrawler full access
User-agent: InteractiveAdvertisingBureau
Allow: /

Whitelist AdsTxtCrawler

Ensure your robots.txt allows InteractiveAdvertisingBureau. Verify requests are genuine by checking the user-agent string and referring to OneTag Limited's documentation.

Traffic & Trends

Cloudflare Radar Traffic

No Cloudflare traffic data available

Google Trends Interest

Links & References

OneTag Limited Documentation

www.onetag.com

Cloudflare Radar: AdsTxtCrawler

radar.cloudflare.com/traffic/bots/adstxtcrawler

Data Sources

This profile was compiled from the following sources:

CloudflareRadar

Last updated: May 26, 2026