Scalable DNS With CoreDNS Plugins: A Deep Dive - Yong Tang, Ivanti & John Belamaric, Google

Scalable DNS With CoreDNS Plugins: A Deep Dive

Speakers: Yong Tang, Ivanti; John Belamaric, Google

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=W3f5Ks0j2Q8

Overview

This technical article delves into the architecture and advanced capabilities of CoreDNS, the default DNS server for Kubernetes. Presented by Yong Tang from Ivanti and John Belamaric from Google at KubeCon EU, the talk highlights CoreDNS's inherent flexibility, its unique plugin-based extensibility model, and recent advancements that address critical scalability challenges in high-performance environments. The speakers underscore CoreDNS's role not just as a Kubernetes component, but as a robust, general-purpose authoritative DNS server written in Go, a memory-safe language that offers significant security advantages over traditional C-based DNS implementations like Bind, which has historically been associated with numerous CVEs.

The core of the presentation revolves around two key aspects: demonstrating the ease with which custom DNS logic can be integrated into CoreDNS through its plugin architecture, and unveiling the multisocket plugin, a pivotal development designed to overcome long-standing QPS (Queries Per Second) limitations. The talk meticulously explains the technical underpinnings of these features, offering practical insights into developing custom plugins and optimizing CoreDNS deployments for maximal performance. It also addresses common operational challenges in Kubernetes DNS, such as the "N-dots problem," providing crucial guidance for defenders and system architects.

Ultimately, this talk is essential for anyone operating Kubernetes clusters, managing large-scale DNS infrastructure, or looking to customize DNS resolution behavior. It provides a comprehensive understanding of CoreDNS's capabilities, its architectural strengths, and practical strategies for enhancing its performance, reliability, and operational efficiency within complex network environments. The emphasis on community contribution and the accessibility of its Go-based codebase further positions CoreDNS as a highly adaptable and future-proof DNS solution.

Background

▶ Watch: Introduction to CoreDNS and its flexibility (0:00)

CoreDNS emerged as a modern alternative to traditional DNS servers, designed specifically to meet the dynamic demands of cloud-native environments, particularly Kubernetes. Unlike conventional DNS servers that often rely on static configuration files, CoreDNS was built from the ground up in Go, a language known for its concurrency features, garbage collection, and crucially, memory safety. This choice of language significantly mitigates a class of vulnerabilities frequently found in C-based applications, such as buffer overflows, which have historically led to numerous security advisories for software like Bind. While CoreDNS can function as a recursive DNS server with appropriate plugins, its primary design intent is that of an authoritative server, serving DNS records for domains it is configured to manage.

A distinguishing characteristic of CoreDNS is its unparalleled flexibility, achieved through a modular plugin architecture. This design allows CoreDNS to integrate seamlessly with various backends, including traditional zone files, the Kubernetes API for dynamic service discovery, and even acting as a caching layer for cloud provider DNS services by reading directly from their APIs. This extensibility means that CoreDNS can be tailored to almost any DNS-related task, from simple name resolution to complex traffic management. The talk highlights that despite its widespread adoption (being the default DNS in Kubernetes), the CoreDNS community is relatively small and welcoming, fostering an environment where new contributors can make significant impacts.

Prior to the advancements discussed in this talk, CoreDNS faced certain operational challenges. One significant issue was a persistent QPS bottleneck, where increasing CPU resources beyond a certain point did not yield proportional improvements in query throughput, and in some cases, even led to a decline. This limitation hindered vertical scaling for very high-traffic scenarios. Another common problem, particularly within Kubernetes, was the "N-dots problem," which leads to excessive DNS queries due to the way client-side resolvers are configured, amplifying the load on DNS servers regardless of their inherent performance. The talk addresses these issues head-on, presenting solutions that significantly enhance CoreDNS's scalability and operational efficiency.

Key Findings

▶ Watch: Demo plugin: Differentiating internal/external DNS requests (2:50)

The talk presented several critical findings and contributions, primarily centered around CoreDNS's extensibility and its enhanced performance capabilities.

Firstly, the core finding reinforced CoreDNS's highly extensible nature through its plugin architecture. It was demonstrated that building custom DNS logic is remarkably straightforward, requiring only a few lines of Go code. This was vividly illustrated by the creation of a split-horizon DNS plugin, which addresses a frequently requested community feature: serving different IP addresses for the same domain name based on whether the request originates from an internal or external network. This highlights that complex, context-aware DNS behaviors can be rapidly implemented and integrated.

Secondly, a major breakthrough was the development and release of the multisocket plugin. This plugin directly addresses a long-standing QPS bottleneck in CoreDNS where, beyond a certain point, adding more CPU cores did not improve, and could even degrade, DNS query throughput. Through experimentation, it was discovered that the bottleneck lay in the single server object's inability to pull requests off the UDP socket fast enough. The multisocket plugin leverages the SO_REUSEPORT kernel feature to allow multiple sockets to listen on the same port (e.g., port 53), with the kernel efficiently distributing incoming packets across these sockets. This innovation enables CoreDNS to achieve near-linear vertical scaling of QPS with increasing CPU cores, significantly enhancing its performance for demanding workloads. The multisocket plugin was released in CoreDNS version 1.12.

Thirdly, an important fix for Kubernetes users was highlighted: the reversion of a change in CoreDNS 1.11.4 that broke the generation of PTR records (reverse DNS records) for Kubernetes service endpoints. CoreDNS 1.12.1 includes this crucial revert, advising users on versions prior to 1.11.4 to upgrade directly to 1.12.1 or later to avoid this issue. This finding underscores the ongoing commitment to stability and compatibility within the Kubernetes ecosystem.

Lastly, the talk provided valuable insights into optimizing CoreDNS deployments in Kubernetes, particularly regarding scaling strategies and mitigating the "N-dots problem." It was suggested to move away from the cluster proportional autoscaler towards CPU utilization-based scaling (e.g., Vertical Pod Autoscaler or Horizontal Pod Autoscaler) for CoreDNS, especially with the multisocket plugin enabled. Furthermore, emphasizing the use of fully qualified domain names (FQDNs) ending with a dot was presented as a critical way to drastically reduce query amplification caused by Kubernetes' default client-side search path configuration, thereby alleviating unnecessary load on DNS servers.

Technical Deep Dive

▶ Watch: Conceptual solution: returning different IPs based on source (4:10)

CoreDNS's architecture is designed for modularity and high performance, built around a central server object and an extensible plugin chain. Internally, a CoreDNS server object is responsible for handling a specific domain and port, corresponding to a single stanza in the Corefile configuration. This server object contains a routine that continuously pulls incoming DNS requests from the underlying UDP socket (or TCP, if configured). These requests are then dispatched to individual go routines, enabling parallel processing of multiple queries concurrently.

The core of CoreDNS's extensibility lies in its plugin chain. When a DNS request arrives, it is handed to the first plugin in a configured sequence. Each plugin can then choose to process the request, modify it, or pass it along to the next plugin in the chain by calling its ServeDNS method. As requests return up the chain, previous plugins also have an opportunity to further modify the response before it is sent back to the client. The order of plugins in the Corefile is crucial for their execution logic, though the overall ordering between plugins is fixed at compile time and requires recompilation to change.

Plugin Development: The Split-Horizon Demo Plugin

The talk provided a practical demonstration of plugin development by showcasing a split-horizon DNS plugin written in Go. This plugin illustrates how to differentiate DNS responses based on the source IP address of the query.

Developing a CoreDNS plugin primarily involves implementing three key functions:

1. init function: This function performs one-time initialization tasks and, critically, registers the plugin's setup function with the Caddy server framework, which CoreDNS uses for its plugin management. For the demo plugin, this was a simple registration of the "demo" plugin with the DNS server type.
2. setup function: This function is called once for each instance of the plugin defined in the Corefile. Its primary role is to parse any plugin-specific configuration directives from the Corefile and add the plugin's handler to the CoreDNS server's plugin chain. In the demo, the configuration was minimal, simply invoking the plugin without complex parameters.
3. ServeDNS function: This is the heart of any CoreDNS plugin, responsible for processing incoming DNS requests. It receives a context and a dns.ResponseWriter, allowing access to the request details (like the query name and source IP) and the ability to write a DNS response. A plugin can decide to respond to the query directly or, if it doesn't handle the specific request (e.g., not its domain), it can pass the request to the next plugin in the chain, enabling a "don't know, ask the next guy" pattern.

For the demo plugin, the ServeDNS function's logic was particularly concise, demonstrated as "eight lines of code." It inspects the source IP of the incoming query. If the IP falls within a predefined internal network range (e.g., 172.x.x.x), it constructs a DNS response with a specific internal IP address (e.g., 1.1.1.1). If the source IP is external, it returns a different external IP (e.g., 8.8.8.8). This simple yet powerful logic allows for dynamic, context-aware DNS resolution.

Configuring such a plugin in the Corefile is straightforward, simply by adding a line like demo within a zone stanza. Building a CoreDNS binary with custom plugins, however, involves static compilation. Developers need to modify CoreDNS's plugin.cfg file to include the new plugin and then compile the entire CoreDNS source code. While this might seem "a little convoluted" as described by the speaker, a Docker command was provided to simplify the build process, resulting in a single CoreDNS binary. The complete demo plugin code, totaling around 80 lines, is available on the CoreDNS GitHub repository.

Multisocket Plugin for Enhanced QPS Scaling

The multisocket plugin was introduced to overcome a critical performance ceiling. Earlier versions of CoreDNS exhibited a non-linear relationship between CPU allocation and QPS, with throughput eventually declining even with more CPU. Investigations revealed that the bottleneck was not in the processing of requests by go routines or the plugin chain, but in the single server object's ability to efficiently retrieve packets from the UDP socket.

An initial experiment involved running two separate CoreDNS instances on different ports and distributing traffic across them, which effectively doubled throughput. This confirmed the single socket as the limiting factor. The solution implemented was to leverage the Linux kernel feature SO_REUSEPORT. This option allows multiple independent sockets to bind to the exact same IP address and port (e.g., 0.0.0.0:53). The kernel then intelligently distributes incoming packets across these multiple sockets.

The multisocket plugin integrates this functionality into CoreDNS. When enabled by adding the multisocket directive to the Corefile, CoreDNS automatically creates a number of sockets equal to the number of CPU cores allocated to its container. This is facilitated by an Uber Go module that automatically sets GOMAXPROCS appropriately in containerized environments. The results were dramatic: performance graphs showed near-linear scaling of QPS up to five CPUs (and beyond, inferred), demonstrating that CoreDNS can now vertically scale efficiently. This means that instead of hitting a QPS wall, CoreDNS can now saturate a 1 Gigabit Ethernet network interface, handling approximately 250,000 QPS (assuming 512-byte UDP packets) before network bandwidth becomes the next bottleneck, or cloud provider limits are hit. This feature, released in CoreDNS 1.12, requires SO_REUSEPORT support in the underlying kernel.

Demo / Proof of Concept

▶ Watch: Deep dive into the serveDNS plugin logic (6:30)

The primary demonstration in the talk was the split-horizon DNS plugin, showcasing the simplicity and power of CoreDNS's extensible architecture. The scenario addressed a common enterprise requirement: serving different IP addresses for the same domain name depending on the origin of the DNS query.

The proof of concept illustrated a network setup where a CoreDNS server sits at the edge, serving both internal and external clients.

• Internal Network: Clients originating from a specific IP range, such as 172.x.x.x (a common private network prefix).
• External Network: Clients originating from any other IP address.

When a DNS query for a specific domain (e.g., example.com) is received:

• If the query comes from an internal client, the custom plugin intercepts it and returns a designated internal IP address (e.g., 1.1.1.1).
• If the query comes from an external client, the plugin returns a different, external IP address (e.g., 8.8.8.8).

The speaker, Yong Tang, emphasized the remarkable ease of implementing this functionality. He noted that the entire plugin, including initialization, configuration parsing, and the core DNS request handling logic, was completed in "one day" and that the critical ServeDNS function contained "only one, two, three, four, five, six, seven, eight lines of code." This highlights the developer-friendly nature of CoreDNS's plugin API, enabling rapid prototyping and deployment of custom DNS behaviors. The plugin's code, available on GitHub under github.com/coredns/coredns/demo, serves as an excellent reference for anyone looking to develop their own CoreDNS plugins. This demonstration effectively proved that CoreDNS is not just a configurable DNS server but a powerful platform for building bespoke DNS solutions.

Defensive Implications

▶ Watch: Configuring and building a custom CoreDNS plugin (8:00)

The advancements and discussions in this CoreDNS talk carry significant implications for system defenders, particularly those managing Kubernetes environments or large-scale DNS infrastructure.

1. Prioritize CoreDNS Upgrades for Kubernetes: The fix for PTR record generation issues, reverted in CoreDNS 1.12.1, is crucial. Defenders should ensure their Kubernetes clusters are running CoreDNS 1.12.1 or a later version. If currently on a version before 1.11.4, it is strongly advised to skip intermediate versions and upgrade directly to 1.12.1 to avoid potential DNS resolution problems, which can impact service discovery and network debugging.
2. Leverage Multisocket for Robustness and Scalability: For high-traffic Kubernetes clusters or standalone CoreDNS deployments, enabling the multisocket plugin is a critical performance and resilience improvement. By allowing CoreDNS to scale QPS linearly with allocated CPU cores, it mitigates a single point of contention (the UDP socket bottleneck) and ensures that DNS services remain responsive under heavy load. This is vital for maintaining application availability and preventing DNS-related outages. Defenders should configure multisocket in their Corefile and monitor CPU utilization to optimize resource allocation.
3. Mitigate the N-dots Problem: The "N-dots problem" is a significant source of unnecessary load on DNS servers in Kubernetes. By default, client-side resolvers (configured via resolve.conf) append a lengthy search path to unqualified domain names, leading to multiple DNS queries for a single lookup. Defenders should:

• Educate developers to use fully qualified domain names (FQDNs) ending with a dot (e.g., myservice.mynamespace.svc.cluster.local.) whenever possible, especially for external services. This immediately reduces query amplification by a factor of up to six for each external lookup.
• Consider client-side configuration adjustments if feasible, though this is often challenging in a dynamic Kubernetes environment.
• Monitor CoreDNS query logs for patterns indicative of the N-dots problem, which can signal excessive load.

1. Optimize CoreDNS Scaling Strategies: The recommendation to shift from cluster proportional autoscaler to CPU utilization-based scaling (e.g., using a Vertical Pod Autoscaler or Horizontal Pod Autoscaler) is a more effective approach, especially with the multisocket plugin. This ensures that CoreDNS pods scale based on actual demand rather than just cluster size, leading to more efficient resource utilization and better performance. Defenders should implement these autoscaling policies.
2. Monitor Memory Consumption: CoreDNS's memory usage directly correlates with the number of services and endpoints in a Kubernetes cluster. In large clusters with many services or headless services, CoreDNS can consume substantial memory. Defenders must set appropriate memory limits for CoreDNS pods to prevent out-of-memory issues and ensure stability. Regularly monitoring memory usage is essential.
3. Implement Node Local DNS: While not a direct CoreDNS feature, the discussion on node local DNS is highly relevant for defensive posture. Running a local caching DNS server on each node:

• Reduces load on central CoreDNS instances.
• Improves latency for pods on that node.
• Crucially, it solves the UDP connection tracking table exhaustion problem. High volumes of short-lived UDP DNS queries can fill the kernel's connection tracking table, leading to dropped packets and DNS resolution failures. Node local DNS typically disables connection tracking for outgoing DNS UDP traffic and upgrades connections to upstream CoreDNS instances to TCP, which is connection-oriented and doesn't suffer from this specific tracking issue. Despite the increased resource footprint (running a CoreDNS instance on every node), the stability and performance benefits often outweigh the costs in high-scale environments. Defenders should evaluate its adoption.

1. Leverage Plugin Extensibility for Custom Security Policies: The ease of developing custom plugins, as demonstrated by the split-horizon example, opens avenues for implementing bespoke security policies directly within CoreDNS. This could include:

• DNS firewalling: Blocking known malicious domains at the DNS level.
• Conditional forwarding: Routing specific queries to security-filtered DNS servers.
• Dynamic response modification: Implementing custom logic to prevent data exfiltration or enforce compliance rules based on query context. The Go language's memory safety also provides a more secure development platform compared to traditional alternatives.

By adopting these defensive strategies, organizations can significantly enhance the security, performance, and reliability of their DNS infrastructure, which is a foundational component of any modern network.

Key Takeaways

• CoreDNS is a highly flexible and extensible DNS server, built in Go for memory safety, and serves as the default DNS in Kubernetes. Its plugin architecture allows for seamless integration with various backends and the implementation of custom DNS logic.
• Custom DNS behaviors are easy to implement with CoreDNS plugins, as demonstrated by the split-horizon DNS plugin, which can differentiate responses based on client origin with minimal Go code (e.g., 8 lines for ServeDNS logic).
• The multisocket plugin significantly improves CoreDNS's QPS scaling, addressing a long-standing bottleneck by leveraging the SO_REUSEPORT kernel feature. This enables near-linear vertical scaling of DNS query throughput with increasing CPU cores, crucial for high-performance environments.
• The Kubernetes "N-dots problem" causes significant DNS query amplification, leading to unnecessary load. Defenders should encourage the use of fully qualified domain names (FQDNs) ending with a dot to mitigate this issue and consider CPU utilization-based autoscaling for CoreDNS.
• Node local DNS is highly recommended for high-scale Kubernetes deployments to improve performance, cache DNS queries, and critically, prevent UDP connection tracking table exhaustion, which can lead to DNS resolution failures.
• CoreDNS 1.12.1 includes a critical fix for PTR record generation in Kubernetes, advising users on older versions (pre-1.11.4) to upgrade directly to this version or later to ensure proper reverse DNS functionality.

About the Speaker(s)

Yong Tang is associated with Ivanti and contributes directly to the CoreDNS project. His involvement suggests a deep technical understanding and hands-on experience with CoreDNS development, particularly in extending its capabilities through plugins, as demonstrated by his lead in the demo plugin development.

John Belamaric is with Google and is a prominent figure in the CoreDNS community. His comprehensive knowledge of CoreDNS's internal architecture, historical challenges, and future directions indicates a significant role in the project, likely as a maintainer or core contributor. He provided extensive context on CoreDNS's evolution, performance optimizations, and operational best practices within Kubernetes.