Spoiler alert: yes. We just haven’t figured out how best to measure it—yet.
There is an old adage that if you can’t measure it, it doesn’t exist. It’s slightly tongue in cheek; there are, of course, plenty of things we know exist but simply have difficulty measuring at scale. But without a firm starting point, it’s very difficult to measure impact. And if you haven’t replicated your metrics consistently, how will you know if you’re improving?
The current level of industry understanding about DNS Abuse is very limited. Currently, alarmingly little is actually known about the prevalence of DNS Abuse within the domain name industry. There are several initiatives that have made great progress in this space, but there is still considerable room for improvement. If DNS Abuse were a disease, the community has only an anecdotal view of the symptoms, without knowing the causes or the opportunities for treatment.
At the Institute, this concerns us because we have one simple mission: reduce DNS Abuse. So how do we know if we’re achieving this? And how does the industry know if their efforts are working? How do we understand the drivers, or figure out which initiatives to scale up?
Measuring things is hard. It’s hard because you need to think about what you’re measuring and why. Defining the issue and drawing lines around the edges of categories isn’t easy. We’ve written before about the need to take a pragmatic approach to the issue of defining DNS Abuse.
We need a reliable, independent, transparent, and sufficiently granular way of understanding DNS Abuse in order to reduce it at the DNS level.
Our purpose for measuring DNS Abuse is to increase our understanding of the problem and bring greater sophistication to community discussions about DNS Abuse. With the ultimate goal of reducing abuse in mind, we want mitigation at the DNS level when it’s effective, quick, simple, precise, proportionate, cost effective, and necessary.
Existing metrics
Unfortunately, what is currently available from a data/metrics perspective doesn’t suit our specific purposes. ICANN’s DAAR project, while technically sound, only provides high-level trends and no actionable data. Similar measurement projects from security companies and reputation block lists have been criticized for having opaque methodology or apparent biases. These lists also tend to be established for the purposes of network blocking rather than action at the DNS level where the margin for error is balanced differently. It’s quite sensible to favor false positives for networks, but much more problematic for DNS level mitigation.
There are still some big parts of the puzzle missing. These lists don’t always differentiate between malicious registrations and compromised websites, which is crucial for our purposes as the mitigation techniques are different. A compromised website involves a victim, who may need to improve their cyber security practices. Suspension of the domain name (without any concurrent engagement with the registrant) is typically not appropriate and could have broad unintended consequences. We’re also interested in more than a list of abuse. We want to understand abuse persistence and whether it has been appropriately mitigated by registrars.
DNSAI Intelligence
To address the measurement conundrum, and advance our mission, we’re launching a new initiative: DNSAI Intelligence, which will focus on two high impact, high volume areas: phishing and malware.
We are committed to sharing metrics that are:
- Credible and Independent – Current DNS Abuse reporting is opaque and commingled with commercial interests. We need an academically robust and independent approach.
- Transparent – The methodology that collects, cleans, and aggregates the data (without processing any personal data) must be as transparent as possible. To the extent that anyone, should they so wish, could replicate the process.
- Accurate and Reliable – The goal of these reports is to enable focused conversations, and to identify opportunities for abuse reduction. The data needs to be of high enough quality to serve as the foundation for meaningful changes to the ecosystem.
This is why we’ve partnered with Kor Labs, run by Maciej Korczynski from Grenoble INP-UGA, to provide this service to the Institute. Korczynski has already contributed much to this field alongside his co-authors by providing the technical analysis for the recent European Commission’s DNS abuse study, the Classification of compromised versus maliciously registered domains (COMAR) study, SADAG – project commissioned by the ICANN Competition, Consumer Trust, and Consumer Choice Review Team, and a paper on Reputation Metrics Design to Improve Intermediary Incentives for Security of TLDs.
This year, we will be producing monthly DNS Abuse reports which will detail where we have found DNS Abuse in the ecosystem, based on the Kor Labs analysis. These will be broken down by the registrar and TLD, will report on phishing and malware, and will identify the composition of compromised websites vs malicious domain registrations. We’ll get an understanding of abuse trends over time, and the impact of growth and abuse by looking at new domain registrations. We’ll report on mitigation of abuse in median hours by registrar as a percentage of domains under management . Finally, we’ll also be looking at the data to explore and learn more about wider abuse trends.
This reporting is intended to shine a new light on DNS Abuse, providing an understanding of what drives it, what helps, and what doesn’t. We’ll be working closely with registrars and registries behind the scenes to share this data directly with them and support them in making changes to impact abuse.
We also expect this will inspire lively discussion in the community. We invite engagement on this topic and want to improve our metrics. If you’re interested in replicating or improving the methodology, we want to hear from you.
Our first set of DNSAI Intelligence reports is expected to be published by September of 2022.
