Tales of a DNS Operator: #1

Welcome to our first blog post on this site! We appreciate, you found your way here. This post marks the introduction of a series of posts which we fittingly dubbed “Tales of a DNS Operator”. Just as the title suggests, we like to use this series to share the tales we have experienced, the insights we have gained and the challenges we have overcome when it comes to operating a global network of open and public DNS resolver.

Introducing newpangea.de Link to this heading

This domain name was acquired years ago with the intention of hosting a handful of honeypots online, which were supposed to gather threat intelligence and Internet anomalies that we planned to provide to the public. As you may have guessed, that was never realized. Instead, I came in touch with the alternative DNS root OpenNIC, which is a supposed to be public, democratic and community-operated set of TLDs, authoritative nameservers and public resolvers.

In the search for a project for my MSc Thesis in computer science, I started to setup and operate one such resolver, and soon as we realized the value one gains from a single public resolver.

This was the moment of birth for this project.

Why DNS? Link to this heading

The domain name system (DNS) is one of the core protocols of today’s Internet. It has been like that for decades. Yet, we see a continous centralization of the domain name space, where global players compete for market shares in order to maximize their competitive advantage over other for- and non-profits.

DNS however was never meant to be centralized. It is designed with scalability and decentralization in mind. Over the decades, DNS has truly proven to be scalable, however it’s is loosing diversity this very moment.

To name a few observations:

In February 2020, Mozilla Firefox activated DNS-over-HTTPS (DoH) for all their US users. Apparently, Cloudflare’s public DNS is the default option. Soon after, Google announces DoH as the default on Chrome. In some other work that is not yet public, we realized that some National Research and Education Networks (NRENs) of developing nations rather announce Google’s or Quad9’s public resolver instead of operating their own recursive resolver.

This really shows you, insights into the DNS are valuable, and global players fight over the market dominance. This has an even worse taste, if you think about countries with loose regulations on personal data. In that case, the DNS data is not only collected to improve the service, but instead is monetized and sold to the highest bidder.

Why Public Resolvers? Link to this heading

As mentioned earlier, the project started with a single resolver which I intentionally exposed to the Internet. It was configured to resolve the regular ICANN, but also OpenNIC top-level domain names (TLDs). Although, the service was online, it was not yet listed anywhere. Yet, within minutes we saw a flood of queries, a handful of scanners, and other anomalies.

First Intel Link to this heading

To this moment, no legitimate traffic was supposed to end up on the server. The initial configuration lacked any best practices like rate-limiting or disabling of ANY requests. Thousands of requests per minute were resolving sl. or cisco.com.. Apparently, those queries appear to generate large DNS responses and will be send back to the requesting IP. Since regular DNS or Do53, i.e. DNS-over-UDP/53 is connectionless, queries can be spoofed. This means, a client sends a query to a DNS server with a different source IP address than it’s own, in order to reflect the DNS response to a target with said IP address.

Ergo, we gained my first threat intelligence insights: We learned which domain names are used for reflection amplification. And also, we learned about IP addresses which are currently under fire of DNS reflection amplification. This was motivation enough to kick-start this project to a larger scale to gain more and more insights.

Surely, we reconfigured my server to throttle requests soon after we discovered this behaviour to not become a burden to the Internet. We plan to public a page soon in which we document the counter-measures we have in place while operating our services. Also, we adhere to the KINDNS best practices.

Why so many public resolvers? Link to this heading

Soon after configuring the first server to be compliant with best practices, we started to add more open resolvers to our fleet of DNS resolvers. As the moment of writing this, we operate 6 public resolvers, with 3 more in the pipeline. Use this overview to find your closests resolvers.

On top of that, we have a central log aggregator, a blacklisting service and an analysis pipeline setup. All of this helps us to further improve our service and to minimize abuse of our services. On top of that, more public resolvers give us more threat intelligence insights and a better understanding of the safe and the not-so-safe parts of the Internet.

The Goal Link to this heading

The goal of this project is to operate public resolvers to the broader community to gain back the decentralized nature of DNS. Not only this, but we also want to prove that you can operate and utilize passive DNS collection, without compromizing a user’s privacy and still be able to gather valuable threat intelligence insights.

Our pipeline is built in such a way, that we abstract IP addresses to the IP prefix and autonomous system number (ASN). We never archive any personally identifiable information (PII) like your IP address. We do not want to have any personally identifiable information in our system. This service is not meant to monetize any user-specific data and thus, we do not even want to store it.

Instead, we operate this service to gather intelligence about the domain name system - a fundamental pillar of the Internet. We do not know, what awaits us with all of this data. Yet, as we’ve pointed out earlier, one can gather valuable insights into the (mis-)use of the domain name system when operating a public resolver. For us, this was reason enough to build a mature and sophisticated network of public resolvers to have our own ground-truth data for research purpose.

Stay Tuned Link to this heading

With our new series of blog posts “Tales of a DNS Operator”, we want to document the progress we make, and the types of DNS abuse we observe. We try to update this website regularly, continously improve our services and share these insights with our community.