SIG Network Intro and Updates - Dan Winship, Nadia Pinaeva, Bowei Du & Daman Arora

SIG Network Intro and Updates

Speakers: Dan Winship, Tech Lead, Red Hat; Nadia Pinaeva, Red Hat; Bowei Du, Broadcom; Daman Arora, Broadcom

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=lBOdQHNNgEU

Overview

The "SIG Network Intro and Updates" session at KubeCon EU provided a comprehensive overview of the crucial work being undertaken by Kubernetes' Special Interest Group (SIG) Network. As the group responsible for defining, implementing, and maintaining the core networking capabilities of Kubernetes, SIG Network plays an indispensable role in ensuring that workloads can communicate effectively and securely within the cluster and with external services. This talk served as both an introduction for new contributors and a detailed update for existing community members and users, highlighting significant advancements, upcoming features, and strategic shifts across key networking APIs and components.

The presentation, delivered by a panel of SIG Network leaders and contributors, covered a broad spectrum of topics, including the evolution and new features of the Gateway API, critical enhancements to kube-proxy, the ongoing development of the Network Policy API (including Admin Network Policy), and the innovative approach to multi-network support. The discussion underscored the group's commitment to improving performance, flexibility, and security, while also addressing long-standing challenges and paving the way for future Kubernetes networking paradigms.

This article delves into the technical specifics of these updates, providing context on their significance for Kubernetes users and operators. From intricate details about TLS connection coalescing in Gateway API to the strategic deprecation of older Ingress controllers, and from advanced connection tracking in kube-proxy to cluster-wide security policies, the insights shared by SIG Network are vital for anyone building, deploying, or securing applications on Kubernetes.

Background

▶ Watch: Introduction to Sig Network and its responsibilities (0:00)

SIG Network's charter is broad and fundamental: it is responsible for the components, interfaces, and APIs that expose networking capabilities to Kubernetes users and workloads. This includes defining new features, maintaining existing ones, and providing reference implementations such as kube-proxy for the Service API. Essentially, almost every aspect of networking within a Kubernetes cluster ultimately falls under the purview of SIG Network. The group operates in a dynamic environment, constantly balancing the need for stability with the demand for new functionalities, improved performance, and enhanced security.

Historically, Kubernetes networking has evolved significantly. The Ingress API, while widely adopted, reached a point of feature stagnation, having been frozen more than five years ago without new capabilities. Its extensibility often relied on controller-specific annotations, leading to fragmented and often non-portable configurations. This limitation paved the way for the Gateway API, a more expressive, extensible, and role-oriented API designed to overcome the shortcomings of Ingress and provide a robust foundation for modern traffic management in Kubernetes. The transition from Ingress to Gateway API represents a major strategic shift, requiring considerable effort to port existing features and encourage ecosystem adoption.

Concurrently, core components like kube-proxy, responsible for implementing the Kubernetes Service concept, continually undergo optimization to handle increasing scale and complexity. This involves moving away from older, less efficient mechanisms and adopting newer kernel features like NFtables for better performance and maintainability. Similarly, network security, traditionally handled by the namespace-scoped Network Policy V1, is expanding to address cluster-wide security requirements through new APIs like Admin Network Policy (ANP). The challenge is to provide powerful, flexible security controls that are intuitive for both application developers and cluster administrators, while ensuring consistent behavior across different Container Network Interface (CNI) implementations.

Key Findings

▶ Watch: Addressing TLS updates and connection coalescing issues (3:00)

The SIG Network update highlighted several critical advancements and strategic directions:

1. Gateway API Maturation and Adoption: Version 1.3 introduces significant features like core support for retry budgets, listener sets for managing large or distributed gateway configurations, and a formal recommendation to return a 421 status code for TLS connection coalescing issues. Percentage-based request mirroring and the policy attachment pattern are also moving to standard status, enhancing traffic management flexibility and API extensibility. A major ecosystem shift is the Ingress-nginx maintainers' transition to Ingate, a Gateway API-focused project, with Ingress-nginx itself entering maintenance mode and slated for archiving in approximately 18 months. This underscores the urgency for the Gateway API to close its current feature gap (currently supporting 35% of Ingress-nginx features, with plans for 55% soon).
2. Kube-proxy Performance and Stability: NFtables is nearing General Availability (GA) in Kubernetes 1.33, offering a more performant backend for kube-proxy (though still opt-in). Significant improvements in connection tracking eliminate reliance on user-space binaries, introducing a conntrack reconciler for robust stale connection cleanup and new metrics for monitoring. The multiple service CIDRs feature has also gone GA, providing scalable and flexible IP range extension.
3. Advanced Traffic Distribution: Prefer-close routing, the third iteration of topology-aware routing, has reached GA in 1.33. New alpha features, prefer-same-zone and prefer-same-node, aim to provide finer-grained control over traffic locality, empowering users to manage load balancing between zones.
4. Enhanced Network Policy APIs: The Admin Network Policy (ANP) and Baseline Admin Network Policy (BANP), currently in Alpha, are being refined for Beta promotion. Key plans include merging ANP and BANP into a single CRD, reworking Port matches to support protocols like ICMP, and addressing user feedback on priority semantics. New implementations from Calico, Kube-OVN, and Antrea, with Cilium on the way, demonstrate growing ecosystem support. Emerging features like tenancy, immutable pod identity match, and FQDN matches (experimental) are also under active development.
5. Multi-Network Evolution: The Multi-Network SIG is abandoning the approach of modifying the core PodSpec for multiple network interfaces. Instead, they are adopting a CRD-based approach leveraging Dynamic Resource Allocations (DRA), treating network interfaces as resources to be assigned to pods. This innovative use of DRA, typically associated with hardware like GPUs, offers a promising path forward.

Technical Deep Dive

▶ Watch: Standardizing request mirroring and policy attachment (4:00)

The technical updates presented by SIG Network touch upon critical areas of Kubernetes networking, from the high-level traffic management provided by Gateway API to the low-level packet handling within kube-proxy and the granular security controls of Network Policy.

Gateway API Advancements

The Gateway API continues its rapid evolution, with version 1.3 bringing several significant features to core support:

• Retry Budgets: A long-requested feature, retry budgets provide a mechanism to limit the number of retries for requests, preventing cascading failures and ensuring system stability under transient error conditions.
• Listener Sets: This feature addresses the challenge of managing a large number of listeners or distributing gateway configurations across multiple namespaces. It allows for the merging of multiple gateways or defining an extensive number of listeners within a single gateway, inspired by use cases requiring over a thousand listeners.
• TLS Connection Coalescing: This thorny issue arises when a client reuses an existing TLS connection, established for one hostname (e.g., bar.example.com via *.example.com), for a subsequent request to a different, more specific hostname (e.g., foo.example.com) that might otherwise map to a different listener with distinct policies. The recommended solution is for implementations to return a 421 "Misdirected Request" status code, preventing traffic from traversing unintended paths and ensuring that gateways warn users about overlapping TLS configurations that could lead to such ambiguities.
• Percentage-Based Request Mirroring: Moving to standard status, this feature allows mirroring a specific percentage of traffic to a different backend. To avoid the limitations of floating-point numbers in Kubernetes API Machinery, it supports fraction or integer values for precise control over traffic distribution.
• Policy Attachment: This generic pattern for extending Kubernetes APIs is crucial for the Gateway API's extensibility. Unlike Ingress, which often relied on annotations, Gateway API uses policies to add additional configuration to routes, services, or gateways. This approach offers better typing, more predictable validation, and a clearer separation of concerns.

To aid the transition and management, two related projects were highlighted:

• Ingress2Gateway: A tool designed to automatically migrate existing Ingress API configurations to the Gateway API, smoothing the path for adoption.
• GatewayCuddle: A kubectl plugin that enhances interaction with Custom Resource Definitions (CRDs) used by Gateway API. It provides functionalities like displaying a full resource graph of connected resources and debugging configurations, for example, showing if a service is referenced by any routes or gateways.

The strategic shift from Ingress-nginx to Ingate is a monumental change. Ingress-nginx, with its 118 annotations, is entering maintenance mode and will be archived. Ingate will focus exclusively on Gateway API support. This transition is challenging because Gateway API currently supports only about 35% of Ingress-nginx's features, with plans to reach 55% soon. SIG Network is actively soliciting feedback via a survey to prioritize the implementation of the most critical Ingress-nginx features in Gateway API, particularly given the lack of telemetry in open-source projects to understand actual usage patterns. The community is encouraged to contribute to close this feature gap.

Kube-proxy and KEP Updates

Kube-proxy, the network proxy for Kubernetes Services, is undergoing continuous refinement:

• NFtables GA: In Kubernetes 1.33, NFtables is slated for General Availability. This modern packet filtering framework offers performance improvements over older iptables backends. However, GA in this context does not mean it will be enabled by default; users will still need to opt-in.
• Connection Tracking Improvements: Significant enhancements have been made to connection tracking. kube-proxy no longer relies on the user-space conntrack binary for cleaning up stale connections, improving performance by avoiding the overhead of forking a process for each entry. A new conntrack reconciler addresses the "best-effort" nature of previous cleanup mechanisms, ensuring more robust stale connection removal. Metrics have also been added to monitor the reconciler's health, total deletions, and time taken.
• Multiple Service CIDRs GA: This feature, now GA, provides a scalable and flexible way to extend Service IP ranges, addressing limitations in previous implementations and fully supporting backward compatibility.
• Traffic Distribution:
• Prefer-close GA: The third iteration of topology-aware routing, prefer-close, has reached GA in 1.33. This mechanism assumes that users will handle more of the balancing between zones themselves, offering a less "clever" but more predictable approach compared to prior attempts.
• Prefer-same-zone and Prefer-same-node Alpha: These new Alpha features, with semantics similar to prefer-close, provide options to prioritize pod scheduling and traffic routing within the same availability zone or even on the same node, respectively.
• API Deprecations and Validations:
• The status.nodeInfo.kubeProxyVersion field is being deprecated and removed.
• DNS search strings are becoming more relaxed, allowing for non-conformant but widely used search strings for pods.
• IP addresses are receiving stricter validation, with a multi-release plan to reject invalid formats (e.g., 0.0.0.0 with padding) for security reasons.
• V1 Endpoints are also slated for deprecation.

Network Policy API Evolution

The Network Policy API continues to expand its capabilities beyond namespace-scoped security:

• Admin Network Policy (ANP) and Baseline Admin Network Policy (BANP): These cluster-scoped APIs, currently in Alpha, empower cluster administrators to set cluster-wide security policies, protecting multiple namespaces simultaneously. For example, an admin can default-deny all traffic across the cluster but specifically allow traffic from a monitoring namespace to all other namespaces.
• Path to Beta: Key improvements planned for Beta promotion include:
• CRD Merging: Consolidating ANP and BANP into a single, unified Custom Resource Definition to simplify management.
• Port Match Rework: Expanding port match capabilities to support additional protocols, with ICMP being a highly requested addition.
• Priority Semantics: Addressing user feedback on the potentially confusing priority field, where the intuitive meaning of lower vs. higher priority for policy application can vary.
• Growing Implementations: The adoption of ANP/BANP is growing, with new implementations from Calico, Kube-OVN, and Antrea joining early adopters like Antrea and OVN-Kubernetes. Cilium is also committed to implementing these APIs.
• Upcoming Features:
• Tenancy: A popular request allowing the definition of a set of namespaces as a tenant, enabling protection from other namespaces in the cluster.
• Immutable Pod Identity Match: Moving towards more secure matching based on Service Accounts rather than dynamic label selectors.
• FQDN Matches (Experimental): Allowing network policies to be defined based on fully qualified domain names (DNS names).
• Policy Assistant: A crucial tool developed by SIG Network Policy API, Policy Assistant helps navigate the complexity of network policies (including ANP). It allows users to:
• Test new policies before applying them to a cluster.
• Understand which policies are affecting traffic.
• Simulate policy behavior using only YAML files, without a running cluster. A demo of this tool was showcased at a previous KubeCon.

Multi-Network Support with DRA

The Multi-Network SIG has pivoted its approach to introducing multiple network interfaces to pods. After extensive discussions, the consensus was to avoid modifying the core PodSpec. Instead, they are now leveraging Dynamic Resource Allocations (DRA). This innovative strategy treats network interfaces as distinct resources, similar to how GPUs or other specialized hardware are allocated. By using DRA, multiple network interfaces can be assigned to a pod through a CRD-based approach, providing a flexible and extensible framework for multi-network configurations without altering core Kubernetes APIs.

Demo / Proof of Concept

▶ Watch: Ingress NGINX shifting to maintenance mode, move to Ingate (6:30)

While the talk itself was primarily an update, several tools and concepts were described that inherently involve demonstration or proof-of-concept capabilities:

• GatewayCuddle: This kubectl plugin was described as being able to "show you a full resource graph of all the connected resources" and "if a service is referenced by any of your routes or gateways." These functionalities imply visual or programmatic demonstrations of how Gateway API resources are interconnected and can be debugged.
• Policy Assistant: This tool, developed by SIG Network Policy API, was explicitly mentioned as having a demo at a previous KubeCon. Its capabilities include testing new policies before application, understanding policy impact on traffic, and simulating policy behavior with YAML files, all of which are practical demonstrations of policy validation and analysis.

These tools serve as practical proof-of-concepts for the complexities they aim to solve, making advanced networking and security configurations more manageable for users.

Defensive Implications

▶ Watch: Survey for prioritizing Ingress NGINX features in Gateway API (7:30)

The updates from SIG Network carry significant defensive implications for Kubernetes cluster operators and security professionals:

1. Enhanced Traffic Control and Security with Gateway API: The Gateway API's structured approach, with policy attachment and clear separation of concerns, provides a more secure and auditable framework for managing ingress traffic compared to annotation-heavy Ingress controllers. Features like retry budgets improve resilience against denial-of-service attempts or backend overload. The formal handling of TLS connection coalescing (returning a 421 error and issuing warnings) helps prevent traffic misdirection and ensures that security policies are applied consistently, mitigating potential bypasses. The shift to Ingate aims for fewer CVEs by moving away from highly extensible but potentially insecure mechanisms like Lua snippets in Ingress-nginx, promoting explicit, safer features.
2. Cluster-Wide Security with Admin Network Policy (ANP): ANP and BANP are game-changers for cluster-wide security. They empower cluster administrators to enforce baseline security postures across all namespaces, preventing lateral movement or unauthorized access by default, and allowing exceptions only where explicitly permitted. This shifts network security from a reactive, namespace-by-namespace effort to a proactive, centralized control plane, significantly reducing the attack surface. Features like immutable pod identity match based on service accounts will further strengthen the integrity of policy enforcement. The Policy Assistant tool is invaluable for defenders, allowing them to test and validate ANP configurations before deployment, ensuring they achieve the intended security posture without disrupting legitimate traffic.
3. Improved Core Network Stability and Validation: kube-proxy enhancements, such as the conntrack reconciler and improved performance with NFtables, contribute to a more stable and reliable network layer, reducing the likelihood of network-related outages or performance degradation that could impact security monitoring or incident response. Stricter IP address validation helps prevent malformed or malicious IP configurations from being introduced into the cluster, closing potential attack vectors.
4. Clarity for Multi-Network Environments: The new DRA-based approach for multi-network support provides a cleaner, more standardized way to attach multiple network interfaces. This clarity is crucial for security, as complex and ad-hoc multi-network configurations can often introduce vulnerabilities or make security policy enforcement challenging. A well-defined API for multi-network interfaces simplifies security auditing and consistent policy application across different network segments.

Overall, SIG Network's work provides defenders with more robust tools, clearer policies, and a more stable foundation for securing Kubernetes environments.

Key Takeaways

• Gateway API is the Future: The Gateway API is rapidly maturing, with new features and a clear path to replace the legacy Ingress API. The Ingress-nginx project's shift to Ingate signals a major ecosystem transition that users should prepare for, utilizing tools like Ingress2Gateway and providing feedback on feature prioritization.
• Cluster-Wide Network Security is Here: Admin Network Policy (ANP) and Baseline Admin Network Policy (BANP) are evolving to provide powerful, centralized controls for cluster administrators to enforce security baselines across all namespaces, moving beyond individual namespace policies.
• Core Network Performance and Reliability Improves: kube-proxy is continually optimized with features like NFtables integration and enhanced connection tracking mechanisms, leading to better performance, stability, and observability of core Kubernetes networking.
• Advanced Traffic Management is Granular: New traffic distribution capabilities, including GA for prefer-close and Alpha features like prefer-same-zone and prefer-same-node, offer fine-grained control over workload traffic locality and load balancing.
• Policy Validation is Crucial: Tools like Policy Assistant are essential for network and security teams to test, validate, and understand the impact of complex network policies before deployment, preventing unintended traffic disruptions or security gaps.
• Multi-Network Adapts with DRA: The strategy for multi-network support has shifted to a CRD-based approach leveraging Dynamic Resource Allocations (DRA), offering a flexible and integrated way to assign multiple network interfaces to pods.

About the Speaker(s)

The "SIG Network Intro and Updates" session was presented by a collaborative team of SIG Network leaders and contributors:

• Dan Winship: A Tech Lead for SIG Network in Kubernetes, Dan works for Red Hat on OpenShift networking. He introduced the SIG's charter and moderated the Q&A segment.
• Nadia Pinaeva: Representing Red Hat, Nadia works on OpenShift networking and is particularly involved with the SIG Network Policy API. She provided detailed updates on the Admin Network Policy and related features.
• Bowei Du: A member of the Kubernetes distribution team at Broadcom, Bowei contributed to the discussion on various SIG Network updates.
• Daman Arora: Also from Broadcom, Daman is part of the Kubernetes distribution team and presented updates on kube-proxy and the multi-network initiative.
• Rob Scott: Working at Google on Gateway-related technologies and other Kubernetes networking aspects, Rob delivered a comprehensive overview of the Gateway API's latest features and strategic direction.
• Adrian: A newer contributor to SIG Network, Adrian works at Salesloft and shared his positive experience joining the SIG, encouraging others to get involved.