Uncharted Waters: Dynamic Resource Allocation for Networking - Miguel Duarte Barroso & Lionel Jouin

Uncharted Waters: Dynamic Resource Allocation for Networking

Speakers: Miguel Duarte Barroso, Software Engineer at Red Hat; Lionel Jouin, Ericsson Software Technology

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=PgCaIeRn6Y

Overview

In the rapidly evolving landscape of cloud-native infrastructure, Kubernetes has established itself as the de facto orchestrator for containerized workloads. However, its historically opinionated and simplistic networking model presents significant challenges for advanced use cases requiring more than a single network interface per pod. This talk, "Uncharted Waters: Dynamic Resource Allocation for Networking," delivered by Miguel Duarte Barroso and Lionel Jouin at KubeCon Europe 2025, addresses these limitations head-on by introducing Dynamic Resource Allocation (DRA) as a Kubernetes-native solution for multi-networking. The speakers delve into how DRA can dynamically provision and manage network interfaces, offering a robust and integrated approach to complex networking requirements that have traditionally relied on out-of-tree solutions.

The core problem tackled is Kubernetes' default "one interface per pod, fully interconnected" paradigm, which, while simple, falls short for scenarios like Network Function Virtualization (NFV), virtual machine (VM) networking (e.g., Qvert), or specialized applications demanding dedicated network segments or multiple interfaces. DRA emerges as a generic framework within Kubernetes, allowing pods to request and utilize a wide array of resources—including network interfaces—with specific characteristics, moving beyond the simplistic integer-based resource requests of older device plugins. This initiative represents a pivotal shift towards a more flexible and powerful Kubernetes networking model, promising to unlock new possibilities for high-performance and sophisticated network-centric applications.

Miguel Duarte Barroso, a software engineer at Red Hat focusing on OpenShift virtualization networking, and Lionel Jouin from Ericsson Software Technology, a key contributor to the CNI and multi-network communities, bring extensive expertise to this critical area. Their presentation not only highlights the motivations and historical context behind the need for enhanced networking but also provides a deep dive into the technical mechanisms of DRA and a compelling live demonstration. For cluster operators, network architects, and application developers grappling with Kubernetes networking constraints, this talk offers a clear roadmap to a future where multi-networking is a first-class citizen within the Kubernetes ecosystem.

Background

▶ Watch: Introduction and agenda overview (0:00)

Kubernetes, by design, champions a remarkably simple and opinionated networking model. Every pod within a cluster is allocated a single network interface, and the expectation is that all pods are fully interconnected and can reach each other without Network Address Translation (NAT) for east-west traffic. While this simplicity facilitates ease of deployment for many applications, it quickly becomes a double-edged sword when more sophisticated networking topologies or resource isolation are required. The default model inherently lacks support for multi-networking, where a pod might require multiple network interfaces, or for specific network characteristics beyond basic connectivity.

For instance, in virtualization use cases, particularly with projects like Qvert (the upstream for OpenShift Virtualization), virtual machines often require dedicated network attachments separate from the cluster's default network. Users frequently opt to omit the default cluster network entirely for their VMs, relying solely on secondary interfaces. Similarly, Network Function Virtualization (NFV) applications, such as firewalls or load balancers, often demand multiple, distinct network interfaces to perform their functions effectively. These requirements cannot be met by Kubernetes' native networking model alone. While Network Policies can be layered on top for micro-segmentation, they are computationally expensive to reconcile and administratively burdensome to write and maintain, making them an inefficient solution for fundamental network topology needs.

To address these shortcomings, the Kubernetes Network Plumbing Working Group introduced Multus CNI as the de facto out-of-tree standard for multi-networking. Multus acts as a CNI meta-plugin, allowing pods to attach to multiple networks by aggregating calls to other CNI plugins. The workflow involves cluster administrators provisioning NetworkAttachmentDefinition (NAD) custom resources (CRs), which essentially serve as a "bag of holding" for CNI configurations. Pod owners then specify desired additional network attachments via a dedicated annotation, providing a JSON-encoded string listing the NADs. While functional, this approach is notoriously error-prone and cumbersome. Typos in the JSON, such as VLAN instead of VLAN ID, can lead to subtle, hard-to-debug issues where pods connect to unintended networks, or outright pod startup failures due to invalid JSON.

Recognizing these limitations and the growing demand for native multi-networking, a subgroup within SIG Network initiated Kubernetes Enhancement Proposal (KEP) 3698. This KEP aimed to bring multi-networking capabilities directly into Kubernetes by proposing a pod.spec.networks array, allowing users to define network attachments in a more structured, typed manner, thereby eliminating the problematic JSON annotations. However, the community feedback was decisive: modifying the core Pod object specification was deemed "taking it way too far." The use case was acknowledged as relevant, but a more generic, Kubernetes-native mechanism for requesting any resource—not just network interfaces—was required.

This rejection highlighted a broader limitation in Kubernetes' resource management: the existing Device Plugin API, introduced in Kubernetes 1.8 (alpha), allows pods to request physical devices (e.g., GPUs, FPGAs) from nodes. However, its resource model is extremely simplistic. Requests are integer-based (e.g., vendor.com/device=5), meaning a pod can only request a count of identical devices. There is no mechanism to specify characteristics of these devices (e.g., "a GPU with 16GB VRAM") nor, crucially, to share a device across multiple pods or claim a virtual partition of a physical device. This inability to express complex resource requirements or enable resource sharing became a significant barrier for advanced networking features like SR-IOV Virtual Functions (VFs) or dedicated bandwidth allocations. The stage was thus set for a more flexible and powerful resource allocation framework: Dynamic Resource Allocation.

Key Findings

▶ Watch: Motivation: virtualization and multi-interface VNFs (3:35)

The rejection of KEP 3698 marked a turning point, not an end, for the pursuit of Kubernetes-native multi-networking. The community recognized the critical need but sought a more generic, extensible solution. This led to the distributed effort across multiple Kubernetes communities, culminating in the development of Dynamic Resource Allocation (DRA). DRA is designed to be a new, Kubernetes-native feature that allows pods to request generic resources, overcoming the limitations of the older Device Plugin API.

A central finding is that DRA effectively conceptualizes network interfaces as a type of resource or device that can be dynamically requested and provisioned for pods. This paradigm shift enables Kubernetes to natively handle complex networking requirements that were previously relegated to out-of-tree solutions. DRA is a rapidly evolving feature, currently in v1beta1 and slated for v1beta2 in Kubernetes 1.33, with a target of General Availability (GA) in Kubernetes 1.34. This aggressive roadmap underscores the community's commitment to integrating advanced resource management directly into the core platform.

Several key Kubernetes Enhancement Proposals (KEPs) and APIs are instrumental in making DRA a reality for networking:

1. KEP 5075 (Virtual Devices and Consumable Capacity): This crucial KEP, championed by Pang from IBM Research, allows users to claim virtual devices on top of existing node resources. This means a pod can request a dedicated bandwidth slice of a physical network interface (e.g., 7 gigabits of a 10 gigabit interface) or a MACVLAN interface with specific characteristics. This capability directly addresses the limitation of older device plugins, enabling shared access and fine-grained resource partitioning.
2. Node Resource Interface (NRI): This new API, integrated into container runtimes like containerd, provides hooks into the pod and container lifecycle events. NRI enables DRA drivers to detect when a pod is being created, allowing them to dynamically configure the requested network interfaces based on the pod's resource claims. This configuration can be orchestrated via CNI plugins or other mechanisms, making the process highly flexible.
3. KEP 487 (Resource Claim Status): Merged and available in alpha since Kubernetes 1.32 and targeting beta in 1.33, KEP 487 allows DRA drivers to report the status of the provisioned devices and network interfaces directly within the Kubernetes API. This status can include critical information such as IP addresses, MAC addresses, interface names, CNI configuration results, and even readiness conditions for individual network interfaces. This granular status reporting is vital for troubleshooting and for enabling network services to leverage secondary IP addresses.

In summary, the key finding is that DRA, supported by these complementary KEPs and APIs, provides a comprehensive and Kubernetes-native framework for multi-networking. It moves beyond simplistic integer requests, allows for complex characteristics, supports virtualized and shared resources, and provides robust mechanisms for configuration and status reporting, marking a significant advancement in Kubernetes' networking capabilities.

Technical Deep Dive

▶ Watch: Multus CNI: current out-of-tree multi-networking solution (4:50)

Dynamic Resource Allocation (DRA) fundamentally re-architects how Kubernetes manages and allocates specialized hardware or software resources to pods. Instead of simple integer counts, DRA introduces a more sophisticated model that allows pods to request resources based on specific characteristics and node requirements. While applicable to GPUs, FPGAs, compute, and storage, its application to network interfaces is particularly transformative.

The DRA workflow is broadly divided into three main phases: scheduling, configuration, and status reporting.

Scheduling with Dynamic Resource Allocation

The scheduling phase is where Kubernetes determines which node can best satisfy a pod's resource claims. This is achieved through a new object called ResourceSlice.

• ResourceSlice: These objects represent the available resources on a specific node. For networking, a ResourceSlice can list all kernel interfaces, physical functions (PFs), and virtual functions (VFs) present on a node, along with their associated attributes (e.g., bandwidth, driver type, capabilities). For instance, a node might have ETH0 and ETH1, each with distinct properties, which would be exposed in its ResourceSlice.
• Pod Resource Request: A pod expresses its need for a network interface not just by name, but by specifying desired characteristics. This might include requesting a MACVLAN interface based on a specific physical Ethernet device (ETH1), or an SR-IOV VF with a guaranteed bandwidth.
• Scheduler Role: The Kubernetes scheduler then evaluates these resource requests against the available ResourceSlice objects across all nodes. Its task is to find a node where all the resources requested by the pod can be provisioned. This capability is crucial for enabling non-uniform clusters, where nodes may have varying hardware configurations.
• Virtual Device Claims (KEP 5075): A significant enhancement in the scheduling aspect of DRA is KEP 5075, which introduces the concept of claiming virtual devices. This allows multiple pods to share a single physical resource by partitioning it into virtualized components. For example, a pod could claim a dedicated bandwidth of 7 Gigabits from a 10 Gigabit ETH1 interface using a MACVLAN. This overcomes the limitation of device plugins where a device could only be exclusively claimed by one pod, enabling efficient resource utilization and advanced network partitioning.

Network Configuration with Dynamic Resource Allocation

Once a pod is scheduled onto a node, the next step is to configure the requested network interfaces. This is facilitated by the Node Resource Interface (NRI).

• Node Resource Interface (NRI): NRI is a new API integrated into container runtimes such as containerd. It provides a standardized mechanism for external plugins or drivers to hook into the lifecycle events of pods and containers. When a pod is being created, NRI notifies the registered DRA drivers.
• DRA Drivers (e.g., CNID driver): A specific DRA driver, such as the CNID driver demonstrated in the talk, is responsible for interpreting the network resource claims made by the pod. Upon receiving a notification from NRI, this driver performs the necessary actions to create and configure the requested network interfaces.
• Configuration Mechanism: The DRA driver can leverage existing tools and APIs for network configuration. Commonly, it would call a CNI plugin to create the desired network interface (e.g., a MACVLAN interface, a VLAN-tagged interface, or an SR-IOV VF) and attach it to the pod's network namespace. The flexibility of NRI means the driver is not limited to CNI; it could integrate with other network configuration tools or directly manipulate kernel network settings. The pod's resource claim template includes the CNI configuration in YAML format, which the driver uses.

Status Reporting with Dynamic Resource Allocation

After configuration, it's essential for the cluster and users to understand the state of the provisioned network interfaces. This is handled by KEP 487 (Resource Claim Status).

• Resource Claim Status (KEP 487): This KEP enables the DRA driver to report detailed status information about the allocated devices and network interfaces back to the Kubernetes API. This status is stored within the ResourceClaim object associated with the pod.
• Reported Data: The reported data is comprehensive and includes:
• IP Addresses: The IP addresses assigned to the newly created network interface.
• MAC Addresses: The MAC address of the interface.
• Interface Name: The name of the interface within the pod's network namespace (e.g., net1).
• Arbitrary Data: This can include the full CNI configuration result, specific driver details, or any other relevant information.
• Conditions and Readiness: Crucially, KEP 487 allows for reporting readiness levels for individual network interfaces. This means a pod could be "ready" in general, but a specific secondary network interface might be "not ready" due to configuration issues. This provides granular visibility into the health of each network attachment.
• Benefits: The detailed status reporting offers two primary advantages:
• Troubleshooting: Operators can quickly diagnose why a network interface failed to configure correctly or why it's not ready, significantly reducing debugging time.
• Network Services: Having IP addresses and other interface details directly available in the Kubernetes API allows higher-level network services (e.g., load balancers, service meshes) to discover and leverage these secondary IP addresses, enabling more sophisticated traffic management and service exposure patterns.

By integrating these three components—scheduling with ResourceSlice and KEP 5075, configuration via NRI, and status reporting with KEP 487—DRA provides a robust and Kubernetes-native framework for dynamic, characteristic-based allocation and management of network interfaces, moving beyond the limitations of previous solutions.

Demo / Proof of Concept

▶ Watch: Challenges: error-prone JSON in Multus configurations (6:00)

The live demonstration provided a clear and practical illustration of Dynamic Resource Allocation (DRA) in action, specifically for provisioning MACVLAN network interfaces. The setup was straightforward yet effective, showcasing how DRA handles node-specific resource availability and integrates with CNI for network configuration.

The demo environment consisted of two Kubernetes worker nodes:

• Kind Worker Node 1: Equipped with two network interfaces, ETH0 and ETH1.
• Kind Worker Node 2: Equipped with only one network interface, ETH0.

The key custom component introduced in the cluster was a CNID driver, which functions as the DRA driver responsible for networking. This driver is designed to configure network interfaces using CNI based on DRA requests. Beyond this, the Kubernetes cluster was standard, without other modifications.

The first step in the demo involved inspecting the ResourceSlice objects created by the CNID driver. As expected, two ResourceSlice objects were shown: one for Worker 1 listing both ETH0 and ETH1 as available network resources, and another for Worker 2 listing only ETH0. This clearly illustrated how DRA exposes the underlying node-specific network device inventory to the Kubernetes control plane.

Next, the speakers deployed a simple application consisting of 15 pods (replicas). The crucial aspect of this deployment was that each pod's ResourceClaimTemplate requested a MACVLAN interface based on ETH1. The ResourceClaimTemplate explicitly contained the full CNI configuration in YAML format (a significant improvement over Multus's JSON-encoded strings) and specified the requirement for ETH1 to be available on the scheduled node.

Upon deployment, a critical observation was made: all 15 pods were scheduled exclusively on Kind Worker Node 1. This perfectly demonstrated DRA's scheduling capabilities, as Worker Node 1 was the only node possessing the ETH1 interface required by the pods' resource claims. Worker Node 2, lacking ETH1, was correctly bypassed by the scheduler. This behavior highlights DRA's ability to enable efficient resource-aware scheduling in non-uniform clusters.

For each scheduled pod, a corresponding ResourceClaim object was created. The speakers then inspected one of these ResourceClaim objects to examine its status. The status field revealed that a new interface named net1 had been configured within the pod. It displayed the assigned IP address, 10.10.1.1/24, and a specific MAC address, 77:76:4C:17.

Finally, to verify the configuration from within the pod itself, ip a command was executed inside one of the running pods. The output confirmed the presence of the net1 interface with the exact IP address and MAC address reported in the ResourceClaim status. This conclusive step validated that DRA, through its CNID driver and CNI integration, successfully provisioned and configured a secondary MACVLAN network interface for the pod in a native Kubernetes way, mirroring the functionality of Multus but with significantly improved robustness and integration. The demo effectively showcased how DRA streamlines the process of attaching pods to secondary networks, making it a first-class Kubernetes operation.

Defensive Implications

▶ Watch: Community feedback: generic resource requests, not pod spec changes (8:00)

The introduction of Dynamic Resource Allocation (DRA) for networking carries several important implications for security and defense strategies within a Kubernetes environment. While DRA primarily enhances operational flexibility and capability, it also reshapes how network resources are managed and secured.

For cluster administrators and security architects, DRA offers a more granular and native way to control network resource allocation, but also demands new considerations:

1. Centralized Network Resource Management: DRA shifts network resource definition from ad-hoc annotations to structured ResourceSlice and ResourceClaimTemplate objects. This provides a more centralized, auditable, and version-controlled approach to defining what network interfaces are available and how they can be requested. This improves the ability to enforce consistent network configurations and reduces the likelihood of misconfigurations leading to security vulnerabilities.
2. Fine-Grained Network Segmentation: By enabling pods to request specific network interfaces (e.g., dedicated VLANs, MACVLANs, or SR-IOV VFs), DRA facilitates true network micro-segmentation at the interface level, rather than solely relying on Layer 3/4 network policies. This allows for stronger isolation between different application tiers or tenants. For instance, sensitive workloads can be placed on interfaces with no default route or with highly restrictive egress/ingress rules applied directly at the network device level, making them inherently more secure.
3. Reduced Error Surface: The move away from error-prone JSON annotations in Multus to typed ResourceClaimTemplates significantly reduces the risk of misconfigurations that could inadvertently expose services or create unintended network paths. This is a direct security benefit, as configuration errors are a common source of vulnerabilities.
4. Enhanced Visibility and Troubleshooting: KEP 487's resource claim status reporting, which includes IPs, MACs, interface names, and readiness conditions for each network attachment, provides unprecedented visibility into the actual network configuration of a pod. This is invaluable for incident response and forensic analysis, allowing security teams to quickly ascertain the network topology of a compromised pod. Granular readiness status can also indicate network misconfigurations or failures that might be exploited.
5. Secure Multi-Tenancy: In multi-tenant environments, DRA can ensure that tenants are allocated isolated network resources (e.g., separate virtual functions or dedicated VLANs) that are strictly managed by the cluster. This prevents one tenant's network configuration from impacting or being visible to another, enhancing tenant isolation.
6. DRA Driver Security: The custom DRA drivers (like the CNID driver in the demo) become critical components. These drivers operate with elevated privileges to configure network interfaces on nodes. Therefore, their security, integrity, and authorization mechanisms are paramount. They must be rigorously vetted, securely deployed, and their access controlled via Kubernetes RBAC. Any vulnerability in a DRA driver could lead to arbitrary network manipulation on the host.
7. Consumable Capacity and Resource Exhaustion: While KEP 5075 enables sharing, it also introduces the need to carefully manage "consumable capacity." Misconfigurations or malicious requests could potentially exhaust shared network resources (e.g., bandwidth on a physical interface), leading to denial-of-service for other workloads. Proper admission control and resource quotas for ResourceClaim objects become essential.

For application developers, DRA simplifies the process of requesting specialized networking, allowing them to focus on application logic while relying on Kubernetes to provide the necessary network infrastructure securely. Developers should be encouraged to utilize these native capabilities, designing applications to leverage multiple, isolated network interfaces for different traffic types (e.g., management, data, control plane), thereby building security into the application's network architecture from the outset.

In essence, DRA empowers defenders with more sophisticated tools to define, control, and monitor network attachments, leading to a more robust and secure Kubernetes networking posture. However, it also necessitates a new understanding of the resource allocation model and diligent management of the DRA drivers and configurations.

Key Takeaways

• Kubernetes' default networking model is inherently limited for advanced use cases, necessitating multi-networking capabilities for scenarios like NFV and virtualization. Traditional out-of-tree solutions like Multus, while effective, are prone to configuration errors due to their reliance on cumbersome JSON annotations.
• Dynamic Resource Allocation (DRA) is emerging as the Kubernetes-native framework to address generic resource requests, including complex network interfaces. It overcomes the limitations of older Device Plugins by allowing pods to request resources based on specific characteristics and supporting virtualized/shared devices.
• DRA facilitates advanced scheduling by leveraging ResourceSlice objects to expose node capabilities and enabling the scheduler to place pods on nodes that can satisfy all specified network resource requirements, crucial for non-uniform clusters.
• New APIs and KEPs are integral to DRA's success: Node Resource Interface (NRI) enables container runtimes to hook into pod lifecycle events for dynamic network configuration by DRA drivers, while KEP 5075 allows for claiming virtual network devices (e.g., specific bandwidth slices or MACVLANs).
• KEP 487 (Resource Claim Status) provides critical visibility by allowing DRA drivers to report detailed status information—including IP/MAC addresses, interface names, and granular readiness conditions—directly into the Kubernetes API, significantly aiding troubleshooting and enabling advanced network services.
• The demo effectively showcased DRA's ability to provision MACVLAN interfaces on specific nodes based on resource availability, simplifying multi-networking and demonstrating a more robust, Kubernetes-native approach compared to previous methods.

About the Speaker(s)

Miguel Duarte Barroso is a Software Engineer currently working for Red Hat. His primary focus is within the OpenShift virtualization networking team, where he is dedicated to pushing advanced virtualization features into OpenShift's Software-Defined Networking (SDN) capabilities. His work involves integrating complex networking requirements typical of virtualized environments directly into the Kubernetes ecosystem.

Lionel Jouin is associated with Ericsson Software Technology. He is a prominent figure in the Kubernetes networking community, actively involved in the CNI (Container Network Interface) community, the multi-network community, and SIG Network. Lionel is specifically involved in the ongoing efforts to integrate multi-networking functionalities into Kubernetes using the Dynamic Resource Allocation (DRA) framework.