Directory/Categories/Advertising & Marketing/AdsTxtCrawler

AdsTxtCrawler

GoodBotVerifiedBOT

OneTag is required to crawl ads.txt files from our publishers to verify the presence of the onetag.com line Operated by OneTag Limited.

Operator website ↗

GoodBot

Assessment

Advertising & Marketing

What is AdsTxtCrawler?

The AdsTxtCrawler, operated by OneTag Limited, is designed to discover and parse ads.txt files on websites. These files are part of the IAB Tech Lab's Authorized Digital Sellers (ads.txt) initiative, which aims to combat ad fraud by allowing publishers to declare authorized sellers of their digital advertising inventory. The crawler identifies these files, extracts the data, and likely uses it to build a comprehensive view of the digital advertising ecosystem, helping to ensure transparency and prevent unauthorized reselling of ad space.

Helpful — Aids in verifying legitimate advertising sources.

Identification

Bot name

AdsTxtCrawler

Operator

OneTag Limited

How we rated this bot

Verified by Cloudflare, which confirms who runs it, with no widespread reports of abuse.

Names itself honestly

Yes

Verified by Cloudflare

Yes

Says it follows robots.txt and opt-outs

Not stated

Ignores site rules

None found

Disguises itself or evades blocks

None found

Overloads sites

None found

How widely people complain

No complaint about this bot in the evidence

Reddit discussion

There is no Reddit discussion of this bot in the evidence

What we read

Why it crawls your site

AdsTxtCrawler crawls the web to locate ads.txt files, which contain a list of authorized sellers for a given website's ad inventory. By collecting and analyzing this data, OneTag Limited can help advertisers and publishers verify the legitimacy of ad sellers and prevent ad fraud. The data gathered by the crawler is used to maintain an up-to-date registry of authorized digital sellers, contributing to a more transparent and secure advertising ecosystem. The bot identifies itself via its user agent and links to the IAB's GitHub repository for the project, allowing easy verification.

Block or allow AdsTxtCrawler

Block it

Add a Disallow rule for InteractiveAdvertisingBureau in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block AdsTxtCrawler from crawling your entire site
User-agent: InteractiveAdvertisingBureau
Disallow: /

# Allow AdsTxtCrawler full access
User-agent: InteractiveAdvertisingBureau
Allow: /

Allow & verify

Ensure your robots.txt allows InteractiveAdvertisingBureau. Verify requests are genuine by checking the user-agent string and referring to OneTag Limited's documentation.