Bots Ad Fraud Future of Media

Inside the fight against 404bot, the ad fraud scheme exploiting ads.txt


By Shawn Lim, Reporter, Asia Pacific

February 25, 2020 | 8 min read

Since the introduction of ads.txt by the IAB Tech Lab more than two years ago, bad actors have invented ways to exploit and subvert a tool that was heralded as a vital way to snuff out ad fraud. The 404bot is the latest issue advertisers must be aware of.

In late 2018, DoubleVerify found fraudsters were launching bot networks to circumvent ads.txt protections, which are designed to allow publishers to list authorized sellers of their inventory. Around the same time, the Integral Ad Science Threat Lab begin monitoring a bot scheme dubbed as 404bot. It noticed a rise in fraudulent activity, specifically domain spoofing, that was likely generated by a single botnet.

The botnet was active from September 2018 to November 2018 but abruptly stopped, coincidentally when BuzzFeed revealed that the FBI, Google, Whiteops and other industry players, had taken down a botnet called 3ve.

3ve, it was later found, was not responsible for the activity of the botnet IAS was monitoring and had unearthed another fraud operation. 404bot meanwhile, was back four months later when traffic spiked from April 2019 to September 2019.

Since then, more than 1.5 billion ads have been affected, according to IAS, mostly video ads. Assuming video ad prices are in a single-digit dollar CPM, fraudsters pocketed more than $15m. A larger payout is likely. It is just a tiny slice of the $30bn of ad fraud estimated to have taken place in 2019.

Publisher’s domains, both high and low profile were hit, and they had one feature in common: an extremely large number of authorized resellers on their ads.txt file.

“The 404bot that IAS identified is a sophisticated form of domain spoofing - falsely misrepresenting a URL so that buyers believe they are getting valid inventory, when in fact it does not exist,” Victoria Chappell, the vice president of marketing for Europe, Middle East, Africa and Asia Pacific at IAS tells The Drum.

“We can only hypothesize the reason for this drop in activity of the botnet, but based on previous observation, we know that 404bot activity could spike again. Publishers should look to audit and update their ads.txt files to mitigate any threat.”

While IAS has spotted the 404bot targeting publishers mostly in the more lucrative digital markets globally, Hemant Menon, the associate director for programmatic at Dentsu Aegis Network feels that APAC will be hit the hardest with losses tripling from $19bn to $56bn and benchmark ad fraud rates hovering anywhere from 1-5%.

“APAC is a mobile-first market and the highest ad fraud rate is on mobile,” he explains to The Drum. “These losses are majorly driven by markets that have seen the highest Internet user growth rates- Indonesia, China and India.”

Ads.txt is compromised - does it have to be?

When ads.txt was created by IAB Tech Lab in 2017, it was intended to increase the transparency of inventory flow in the online advertising ecosystem.

It remains an effective way to vet supply coming in via automated platforms as it contains the domain and seller ID of the publisher. Its flaw is that it is reliant on trust on the intermediaries in the supply chain.

According to Menon, this has resulted in domain spoofing. There have also been cases of fake and blank ads.txt. Some exchanges will only work with publishers that have ads.txt but won't vet the content of their logs. This has led to a rise in ad fraud for the buyers as they thought these were “authorized sellers”.

Other cases include ads.txt with deliberate errors that are done deliberately and usually for nefarious reasons. Ads.txt syndication is a type of fraud that occurs when smaller publishers who are not able to sell their inventory “rent” ads.txt files from larger tier-one publishers and monetize their traffic.

“Despite widespread adoption of ads.txt, however, our data suggest that domain spoofing still exists and can be quite prevalent,” explains Chappell.

She adds: “We wondered if publishers were not properly vetting resellers, or if they were simply using ads.txt on their websites as a formality. The former, if true, defeat the core purpose of ads.txt’s existence.”

Damon Reeve, chief executive officer at The Ozone Project believes that ads.txt has had a significant role in reducing fraud but is aware fraud still exists. He's not surprised to hear that the 404bot has been exploiting ads.txt.

“Unfortunately no system or solution is perfect. If there are improvements that can be made then we should work towards implementing them, rather than saying it's not working,” he tells The Drum.

Jason Barnes, chief revenue officer for APAC at PubMatic, agrees, adding that as long as there are dollars to be made, bad actors will continue to find a way. He says, there are almost zero risks from law enforcement, and as soon as one threat is discovered, another botnet like 404bot rises to siphon away ad dollars.

The industry is unlikely to fully eradicate ad fraud, but it can get better at fighting and preventing it.

As the proliferation of digital media continues, there will be more ways for consumers to interact and more ways to advertise, each presents a new opportunity for fraudsters.

“We think about ad fraud as an ongoing fight. The digital industry is always evolving at a rapid pace, this means fraudsters, and fraud detection companies have to go through a huge amount of fast-paced change also,” he adds.

How can brands and publishers protect themselves?

Brands and publishers need to work with transparent supply chains, reputable supply partners, and know what ads are appearing - and where.

Pubmatic's Barnes says that the 404bot proves ads.txt vulnerabilities. However, initiatives like ads.txt, sellers.json and Supply Chain Object discourage fraudulent activity by allowing publishers to control who sells their inventory and providing clarity on all purchasers in the supply chain.

“Publishers need to ensure those ads.txt files are regularly updated and they are only making their inventory available to reputable and transparent supply partners. They should be taking steps to ensure that there is no on-selling of the inventory,” he explains.

Cadi Jones, commercial director for EMEA at Beeswax points to a further sister IAB initiative to ads.txt called ads.cert, which enables the sellers to sign bid requests cryptographically.

She says effectively this should validate that the information passed in the bid request is accurate, and not tampered with in any process through the bid request or response mechanism.

“Generally, we encourage our customers to have a direct relationship with the supply partners that they are buying from, both an SSP relationship and even a publisher relationship,” she explains.

“This allows any issues from the buy-side, or the sell side to be flagged and addressed rapidly. The focus on supply path optimisation or demand path optimisation allows for greater transparency and aligned goals for both the publisher and the advertisers.”

What's next?

With the 404bot shows no signs of slowing, Menon says the next step of protection should see brands and publishers adopt Open RTB Supply Chain Object and sellers.json respectively which will help both sides ensure transparency and authenticity of the transaction.

He also urges both suppliers and buyers to come together to discuss further solutions to eradicate fraud transactions. Ad fraud profit needs a place in the chain to be collected, points out Barnes. It is important for marketers to understand the supply paths from which inventory is sourced.

Marketers should be familiar with the apps and domains where their ads are running, and be able to confirm the reputation of companies participating in the supply chain. “If you take out bad actors, the probability of ad fraud is greatly reduced,” he says.

Brands and publishers should also beware of chasing clicks as the primary metric for campaign success, adds Barnes, as there is no better way to encourage fraudulent activity than for marketers and agencies to demand high click-through rates or volumes as buying will inevitably move towards scaled cheap inventory that is often non-transparent.

In the fight against ad fraud and the fraudsters that lie in wait, it is vital that brands are working with verification companies with sophisticated detection techniques and to continuously audit and update ads.txt files.

That will go a long way in making sure ads are served by real publishers, shown to real people and are reaching the right target audiences.

"Ads.txt is still crucial, however," says Amir Malik, a digital marketing expert at Accenture Interactive. "As the attacks on the programmatic ecosystem become stronger, there’s a clearer need to consolidate. As well as the more long-term need for human intervention and talent to support clients in their digital marketing journey."

Additional reporting by John McCarthy

Bots Ad Fraud Future of Media

More from Bots

View all


Industry insights

View all
Add your own content +