Making the Leap: What Gateway API Needs To Support Ingress-NGINX Users - Rob Scott & James Strong

Making the Leap: What Gateway API Needs To Support Ingress-NGINX Users

Speakers: Rob Scott, Kubernetes Networking, Google; James Strong, Solutions Architect, Isovalent (now Cisco)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=EeegJKZ_4g0

Overview

This talk, presented by Rob Scott of Google and James Strong of Isovalent (now Cisco), addresses the critical transition facing a vast segment of the Kubernetes community: the migration of users from the highly prevalent Ingress-NGINX controller to the burgeoning Gateway API. As a maintainer of the Gateway API and a seasoned user of Ingress-NGINX, Rob Scott, alongside James Strong, details the ambitious, yet necessary, shift being undertaken by the Kubernetes SIG Network community. The core message revolves around standardizing Kubernetes networking primitives and consolidating development efforts into the Gateway API, which is positioned as the successor to the original Ingress API.

The significance of this talk cannot be overstated. Ingress-NGINX has long been one of the most popular ingress controllers, reportedly used by 40% of Kubernetes clusters, thanks to its robust feature set and extensive customization options, primarily delivered through annotations. However, this very strength has become a formidable challenge for maintainers and a barrier to broader API standardization. The speakers meticulously outline the current state of feature parity, the complexities involved in migration, and the strategic initiatives underway to bridge the gap, including the creation of a new Gateway API implementation called Ingate and dedicated efforts within the Gateway API project to accommodate Ingress-NGINX users.

The talk serves as both an informative update and a direct call to action for the community. It underscores the technical hurdles, the need for user feedback to prioritize features, and the long-term vision for a more unified, extensible, and maintainable Kubernetes networking ecosystem. For organizations heavily invested in Ingress-NGINX, understanding this transition is not merely about adopting a new API but about securing a sustainable path for their future Kubernetes deployments.

Background

▶ Watch: Introduction: Ingress-NGINX users migrating to Gateway API (0:00)

The Kubernetes ecosystem has long relied on the Ingress API for exposing services, with Ingress-NGINX emerging as a dominant implementation. According to data cited from Wiz, Ingress-NGINX is deployed in approximately 40% of Kubernetes clusters, a testament to its widespread adoption and perceived utility. Its popularity stems from its ability to extend the basic Ingress API with a rich set of features, primarily through 118 unique annotations and extensive ConfigMap options. This flexibility, while powerful, has inadvertently created a parallel, de facto API for ingress configuration, making it difficult to maintain and standardize across the broader Kubernetes landscape. The speakers highlighted the significant burden on maintainers, citing the difficulty of supporting such a massive and diverse feature set, especially when dealing with security vulnerabilities (CVEs) and the inherent complexity of managing features implemented via annotations, some of which even involve Lua scripting.

In parallel, the Gateway API has been steadily maturing, officially graduating to GA (General Availability) in 2023. Conceived initially as "Ingress v2," the Gateway API was designed from the ground up to address the limitations of the original Ingress API, offering a more expressive, role-oriented, and extensible framework for Kubernetes networking. It is a superset of the Ingress API, boasting over 30 implementations and a highly collaborative development model involving hundreds of contributors. The core motivation behind the Gateway API was to standardize network configuration, enable advanced use cases, and provide a portable API that works across diverse environments and proxy implementations.

The confluence of Ingress-NGINX's immense popularity and the Gateway API's emergence presented a critical juncture for the SIG Network community. The decision was made to shift focus from maintaining and extending Ingress-NGINX to encouraging migration to the Gateway API. This transition involves moving Ingress-NGINX into maintenance mode, with an approximate 18-month roadmap towards this shift. To facilitate this, the Ingress-NGINX maintainers are developing a new project called Ingate, which will be a Gateway API implementation specifically designed to support the most critical features used by Ingress-NGINX users. The rationale for this strategic pivot is multifaceted: to standardize all SIG Network projects on a unified API, to address the long-frozen Ingress API (which hasn't seen new features in over five years), to improve the security posture by better validating user input (preventing CVEs), to enable other Gateway API implementations to benefit from shared features, and critically, to manage the limited bandwidth of open-source contributors who are stretched thin across multiple, sometimes overlapping, projects.

Key Findings

▶ Watch: Ingress-NGINX's 118 annotations and maintenance challenges (2:00)

The central revelation of the talk is the significant feature parity gap that currently exists between the extensive capabilities offered by Ingress-NGINX and the current state of the Gateway API. The speakers presented a stark visual representation of this challenge:

• Current Support: As of Gateway API v1.2, only approximately 35% of the functionality provided by Ingress-NGINX's 118 annotations can be directly implemented.
• Soon-to-be Supported: An additional 20% (bringing the total to 55%) is "likely soon" to be supported, with active work and PRs in progress within the Gateway API.
• No Current Plans: A substantial 45% of Ingress-NGINX features currently have no concrete plans for implementation within the Gateway API. This "red slice" of the pie chart represents a critical barrier to migration for many users.

This gap is further exacerbated by the fact that many of Ingress-NGINX's powerful features, such as external authentication and advanced rate limiting, are either entirely missing or still under active discussion in the Gateway API. While a request for ModSecurity (Web Application Firewall) integration has been initiated, it highlights that even fundamental security features require community drive to be incorporated. The speakers emphasized that the community's feedback through a dedicated survey is paramount to prioritizing which of these 118 annotations and their underlying functionalities will be targeted for implementation in the Gateway API.

Conversely, the talk also highlighted the new capabilities that Gateway API unlocks, which are often difficult or impossible to achieve with Ingress-NGINX's annotation-based model:

• Advanced Traffic Matching: The ability to match traffic based on HTTP headers, query parameters, and methods.
• Request/Response Modification: Granular control over modifying request and response headers.
• Cross-Namespace References: Securely referencing services or routes across different namespaces, enabling more complex multi-tenant or distributed application architectures.
• gRPC Routing: Native support for routing gRPC traffic, crucial for modern microservices.
• Advanced Traffic Splitting: More sophisticated traffic management scenarios beyond basic canary deployments.
• Service Mesh Integration: A unified API for both ingress and service mesh, simplifying configuration for environments using meshes.
• Inference Gateway Extension: A novel extension for transforming any Gateway into an inference gateway, specifically designed to aid in routing for Large Language Models (LLMs) within a cluster, signaling the API's forward-looking innovation.

Beyond features, the talk acknowledged that the migration challenge extends to user experience and operational complexity. Gateway API, by design, introduces more resources (GatewayClass, Gateway, HTTPRoute) compared to the single Ingress resource. It leverages Custom Resource Definitions (CRDs), which, while enabling rapid feature delivery independent of Kubernetes releases, introduce challenges in CRD management (though some providers like GKE manage them) and a less-than-ideal kubectl experience for describing and interacting with CRDs. These factors contribute to a perception of increased complexity, especially for "Hello World" scenarios.

Technical Deep Dive

▶ Watch: Announcement: Shifting to Ingate, Ingress-NGINX maintenance mode (4:00)

The technical core of the migration challenge lies in the architectural differences and feature implementation paradigms of Ingress-NGINX and Gateway API. Ingress-NGINX, while powerful, has historically relied on a sprawling set of 118 annotations to expose its vast functionality. These annotations, coupled with ConfigMap settings and 68 command-line flags, allow users to customize nearly every aspect of NGINX behavior. The underlying implementation often involves Lua scripting, which, as James Strong noted, can be a maintenance burden due to its specialized nature and the difficulty in ensuring secure, validated user input, leading to past CVEs. Furthermore, Ingress-NGINX manages 43 dependencies across four architectures and supports three Helm versions, requiring continuous testing (over 100 end-to-end tests) and significant maintainer effort.

Gateway API, in contrast, adopts a more structured, role-oriented approach with distinct resources:

• GatewayClass: Defines a class of Gateways, indicating the controller responsible for implementing it.
• Gateway: Represents the actual proxy instance, defining listeners (ports, protocols) and attached certificates.
• HTTPRoute (and GRPCRoute, TCPRoute, TLSRoute, UDPRoute): Define routing rules for specific protocols, attaching to a Gateway.

These resources are all implemented as CRDs. This design choice offers a key advantage: new Gateway API versions and features can be released independently of Kubernetes core, allowing for rapid innovation. Users can install new versions as long as they are running one of the five most recent Kubernetes versions, eliminating the need to wait for a full Kubernetes upgrade. However, the CRD approach also presents operational challenges, as users typically have to manage CRD lifecycle themselves (though some cloud providers like GKE are starting to manage them).

To address the perceived complexity and user experience hurdles, the Gateway API community is actively exploring several enhancements:

• Simplifying HTTPRoute: Making HTTPRoute more self-sufficient, potentially allowing it to function similarly to an Ingress resource in simple cases, reducing the need to define a Gateway explicitly for basic use cases. The goal is to make the "Hello World" experience as straightforward as Ingress.
• Default Gateway: Introducing the concept of a default Gateway to abstract away the initial setup, allowing users to focus solely on HTTPRoute definitions for common scenarios.
• kubectl Experience Improvements: Recognizing that kubectl describe for CRDs often yields raw YAML output without helpful contextual information, efforts are underway to improve upstream kubectl to better handle CRDs, including displaying connected resources and status conditions more intuitively.

A significant technical contribution to improving the Gateway API user experience is gatewayctl. This specialized command-line tool acts as a "drop-in replacement" for kubectl for Gateway API resources. gatewayctl is designed to overcome kubectl's current limitations with CRDs by:

• Showing Connected Resources: Easily visualize how different Gateway API resources (Gateways, Routes, Services) are linked. For example, gatewayctl get service --routes can show which routes point to a specific service.
• Printing Resource Graphs: Generate a graphical representation of the entire Gateway API configuration, illustrating the relationships between objects.
• Analyzing Configuration: Identify problems, misconfigurations, or broken references within the Gateway API setup.
• Detecting Policies and Extensions: Group connected policies and extensions with their core Gateway API resources to provide a holistic view.
• Enhanced describe Output: Provide a more human-readable and contextual describe output for Gateway API resources, indicating attached routes, backends, and more. For instance, describing a service would show routes pointing to it, aiding in dependency analysis before deletion.

The migration path itself is also a technical challenge. The ingress2gateway project from Kubernetes SIG aims to automate the conversion of Ingress manifests to Gateway API resources. However, it currently supports only 6 out of the 118 Ingress-NGINX annotations, underscoring the vast amount of work still required to make this a viable automated migration path.

To bridge the feature parity gap, the Gateway API project is implementing a strategic change: adding two temporary feature slots per release, exclusively dedicated to Ingress-NGINX compatibility. With a target of at least three releases in 2025 and three in 2026, this translates to approximately 12 dedicated feature categories. The speakers emphasized that these are "categories" of features, not individual annotations, meaning each slot could cover multiple annotations. These new features will be subject to all existing Gateway API criteria, ensuring security, maintainability, and, crucially, portability across different proxy implementations (e.g., NGINX, Envoy, Cilium). Features that are proxy-specific or cannot be securely implemented will not be accepted. This "special treatment" is justified as providing a safe migration path for users of an officially archived SIG Network project.

The timeline for Ingress-NGINX's transition to maintenance mode is approximate, with the goal of having a stable Ingate (the new Gateway API implementation) for testing by the Atlanta KubeCon. The Ingress-NGINX project itself plans one more minor release (v1.13) for new features and will continue monthly patch releases, supporting Kubernetes versions until 2027, before being formally archived.

Demo / Proof of Concept

▶ Watch: Why the shift? Standardizing on Gateway API (5:30)

While the talk did not feature a live, interactive coding demo, the speakers provided a compelling demonstration of gatewayctl's capabilities through illustrative command outputs and resource graphs. These "proofs of concept" showcased how gatewayctl significantly enhances the user experience and operational visibility for Gateway API configurations, addressing some of the inherent complexities of working with CRDs.

The gatewayctl demonstrations highlighted several key functionalities:

1. Showing Connected Resources:

• Service to Routes: The command gatewayctl get service my-service --routes was shown to display all HTTPRoute resources that are configured to route traffic to my-service. This is invaluable for understanding dependencies and impact analysis, for example, before deleting a service.
• Gateway to Routes: Similarly, gatewayctl get gateway my-gateway --routes would list all HTTPRoute objects attached to a specific Gateway. This helps operators understand the full scope of traffic being managed by a particular proxy instance.
• GatewayClass to Gateways: gatewayctl get gatewayclass my-class --gateways illustrates which Gateway resources are using a particular GatewayClass, providing insight into the adoption and usage of different Gateway API implementations.

1. Enhanced describe Output:

• The speakers demonstrated how gatewayctl describe gateway my-gateway provides a much richer and more contextual output than kubectl describe for CRDs. Instead of raw YAML, it explicitly lists attached routes, their backend services, and other relevant configurations, making it easier to grasp the Gateway's overall function and its connections within the cluster.
• For a service, gatewayctl describe service my-service would show not only the service details but also "pointing up here are the routes that are pointing to my service," explicitly answering the question of who is consuming that service.

1. Printing Resource Graphs:

• A visual representation of a resource graph generated by gatewayctl was presented, illustrating how different Gateway API objects (GatewayClass, Gateway, HTTPRoute, Service) interlink. This graphical output helps users visualize the entire networking topology managed by Gateway API, simplifying complex configurations and troubleshooting.

These demonstrations effectively conveyed gatewayctl's role in mitigating the learning curve and operational friction associated with the multi-resource nature of Gateway API. By providing immediate, clear insights into resource relationships and configuration status, gatewayctl acts as a crucial bridge, making the transition to Gateway API a more manageable and less daunting task for engineers accustomed to the simpler, albeit less expressive, Ingress API. The speakers emphasized that this was just 0.1 of gatewayctl, indicating a strong commitment to further enhancing the user experience.

Defensive Implications

▶ Watch: Feature parity challenge: Only 35% of annotations supported (7:00)

The impending shift from Ingress-NGINX to Gateway API carries significant defensive implications for organizations running Kubernetes. Proactive planning and strategic adoption are crucial to maintain a robust security posture and efficient operations.

1. Proactive Migration Strategy: Organizations heavily reliant on Ingress-NGINX must begin planning their migration strategy immediately. This involves inventorying all existing Ingress-NGINX configurations, especially those leveraging the 118 annotations. A thorough audit will identify which critical features fall into the "no plans for implementation" category within Gateway API, necessitating engagement with the SIG Network community.
2. Security Posture Improvement: The move away from annotation-driven extensions in Ingress-NGINX to the more structured and validated Gateway API design can inherently improve security. Annotations, being free-form strings, are prone to misconfiguration and can introduce vulnerabilities if not properly validated and sanitized, as evidenced by past CVEs discussed in the talk. Gateway API's explicit resource definitions and stricter schema validation reduce this attack surface. Defenders should view this as an opportunity to standardize secure ingress configurations.
3. Enhanced Visibility and Troubleshooting with gatewayctl: The gatewayctl tool, demonstrated in the talk, is a vital asset for defenders. Its ability to show connected resources, print resource graphs, and analyze configurations provides unprecedented visibility into the ingress layer. This is critical for:

• Impact Analysis: Before making changes or decommissioning a service, gatewayctl can quickly identify which routes are pointing to it, preventing unintended service disruptions.
• Security Audits: Easily visualize the entire traffic flow and identify potential misconfigurations or unauthorized routes.
• Incident Response: Rapidly understand the networking topology during an incident to pinpoint the source or scope of an attack.

1. Community Engagement for Feature Prioritization: Given that 45% of Ingress-NGINX features currently lack a migration path, defenders must actively participate in the Gateway API community survey. Prioritizing essential security features (e.g., advanced WAF integration, specific authentication mechanisms, fine-grained rate limiting) ensures that critical defensive capabilities are included in the API's roadmap. Failure to engage risks losing vital functionality.
2. Understanding Ingress-NGINX Maintenance Mode: Organizations must be aware that Ingress-NGINX is entering maintenance mode, with only one more minor release (v1.13) planned for new features and patch support extending to Kubernetes versions until 2027. Relying on Ingress-NGINX beyond this timeframe, especially for new deployments, will expose them to increasing security risks as new vulnerabilities may not be addressed. This hard deadline should accelerate migration efforts.
3. Embracing Portability for Resiliency: Gateway API's emphasis on portability across different proxy implementations (NGINX, Envoy, Cilium, etc.) offers a defensive advantage. It allows organizations to switch underlying proxy technologies without re-architecting their ingress configurations, providing greater flexibility and resiliency against vulnerabilities specific to a single proxy.
4. RBAC Considerations: The Gateway API introduces a more granular, role-oriented RBAC (Role-Based Access Control) model compared to Ingress. Defenders should carefully design their RBAC policies around GatewayClass, Gateway, and Route resources to enforce least privilege and prevent unauthorized modifications to networking configurations.

In conclusion, the migration to Gateway API, while challenging, presents a strategic opportunity for defenders to enhance their Kubernetes networking security, improve operational visibility, and future-proof their infrastructure against evolving threats and technological shifts. Ignoring this transition is not an option for security-conscious organizations.

Key Takeaways

• Ingress-NGINX is transitioning to maintenance mode, with an approximate 18-month roadmap towards archival. Its maintainers are developing Ingate, a new Gateway API implementation, as its successor.
• A significant feature parity gap exists between Ingress-NGINX's 118 annotations and Gateway API. Currently, only 35% of Ingress-NGINX functionality is directly supported, with 55% expected "soon," leaving a substantial 45% without a clear migration path.
• Community feedback through the provided survey is crucial for prioritizing which Ingress-NGINX features are implemented in Gateway API. This will directly influence the development roadmap for the next 18-24 months.
• Gateway API offers advanced, standardized networking capabilities beyond Ingress-NGINX's reach, including header/query param matching, cross-namespace routing, gRPC support, and innovative extensions like inference routing for LLMs.
• The gatewayctl CLI tool is being developed to enhance the user experience with Gateway API, providing better visibility into connected resources, configuration analysis, and improved describe output, addressing the perceived complexity of CRDs.
• The Gateway API project is dedicating temporary feature slots in upcoming releases specifically to close critical Ingress-NGINX feature gaps, focusing on portable and securely implementable functionalities to ease the migration burden.
• Organizations should proactively plan their migration strategy, audit existing Ingress-NGINX configurations, and engage with the Gateway API community to ensure their critical use cases are supported in the future.

About the Speaker(s)

Rob Scott works at Google, focusing on Kubernetes networking. He is a primary maintainer of the Gateway API and has extensive experience using Ingress-NGINX heavily in production across dozens of clusters, expressing gratitude for its past utility. His dual perspective as both a developer of the future API and a user of the current dominant solution provides unique insights into the migration challenges.

James Strong is a Solutions Architect at Isovalent (now Cisco). In his role, he assists organizations in implementing Kubernetes networking solutions, particularly with Cilium, and securing their clusters using Tetragon. His practical experience in deploying and securing Kubernetes environments provides a valuable, real-world perspective on the operational implications of network API transitions.