Directory/Categories/Advertising & Marketing/AdsTxtCrawler

AdsTxtCrawler

Name: AdsTxtCrawler
Author: OneTag Limited

GoodBotVerifiedBOT

OneTag is required to crawl ads.txt files from our publishers to verify the presence of the onetag.com line Operated by OneTag Limited.

Operator website ↗

GoodBot

Assessment

Advertising & Marketing

What is AdsTxtCrawler?

The AdsTxtCrawler, operated by OneTag Limited, is designed to discover and parse ads.txt files on websites. These files are part of the IAB Tech Lab's Authorized Digital Sellers (ads.txt) initiative, which aims to combat ad fraud by allowing publishers to declare authorized sellers of their digital advertising inventory. The crawler identifies these files, extracts the data, and likely uses it to build a comprehensive view of the digital advertising ecosystem, helping to ensure transparency and prevent unauthorized reselling of ad space.

Helpful — Aids in verifying legitimate advertising sources.

Identification

Bot name

AdsTxtCrawler

Operator

OneTag Limited

Why it crawls your site

AdsTxtCrawler crawls the web to locate ads.txt files, which contain a list of authorized sellers for a given website's ad inventory. By collecting and analyzing this data, OneTag Limited can help advertisers and publishers verify the legitimacy of ad sellers and prevent ad fraud. The data gathered by the crawler is used to maintain an up-to-date registry of authorized digital sellers, contributing to a more transparent and secure advertising ecosystem. The bot identifies itself via its user agent and links to the IAB's GitHub repository for the project, allowing easy verification.

Block or allow AdsTxtCrawler

Block it

Add a Disallow rule for InteractiveAdvertisingBureau in your robots.txt file. You can also block at the server level using your web server configuration or CDN firewall rules to filter requests matching the user-agent string.

# Block AdsTxtCrawler from crawling your entire site
User-agent: InteractiveAdvertisingBureau
Disallow: /

# Allow AdsTxtCrawler full access
User-agent: InteractiveAdvertisingBureau
Allow: /

Allow & verify

Ensure your robots.txt allows InteractiveAdvertisingBureau. Verify requests are genuine by checking the user-agent string and referring to OneTag Limited's documentation.