Navigating the Inevitable: Kubernetes Breaking Changes Behind the Scenes - Marko Mudrinić

Navigating the Inevitable: Kubernetes Breaking Changes Behind the Scenes

Speakers: Marko Mudrinić, Senior Software Engineer, Kubermatic

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=lQFUarM_GXo

Overview

In this insightful KubeCon EU session, Marko Mudrinić, a Senior Software Engineer at Kubermatic and a prominent figure in the Kubernetes community, including his roles as SIG K-infra Tech Lead and sub-project lead for Kubernetes Release Engineering, delved into the often-dreaded topic of Kubernetes breaking changes. The talk, titled "Navigating the Inevitable," aimed to demystify how core Kubernetes handles these changes, why they are necessary, and how users can effectively prepare for and mitigate their impact. Mudrinić emphasized that while breaking changes are an unavoidable aspect of any evolving software project, Kubernetes employs a structured, policy-driven approach to minimize disruption, especially for General Availability (GA) features.

The core message underscored that most Kubernetes breaking changes come with significant advance notice and provide clear mitigation paths, requiring user action rather than outright feature removal without replacement. However, Mudrinić also highlighted a critical distinction: the infrastructure supporting Kubernetes (such as image registries and package repositories) operates under different rules, with no guarantees and potential for abrupt, short-notice changes. This talk is crucial for anyone operating Kubernetes clusters, from cluster administrators to application developers, offering a behind-the-scenes look at the project's philosophy, processes, and actionable strategies for navigating the continuous evolution of the world's leading container orchestration platform.

Background

▶ Watch: Introduction and defining a breaking change (0:00)

To understand Kubernetes breaking changes, it's essential to first define what constitutes a "breaking change." Mudrinić adopted a user-centric definition: "a change that's not backwards compatible or requires a user action upon upgrading." He categorized these into two types: those with a mitigation path (where the feature still exists but requires adaptation) and those without (where the feature is entirely removed with no replacement). Fortunately, the vast majority of Kubernetes breaking changes fall into the former category, allowing users to adapt rather than being left stranded.

Kubernetes officially classifies these significant alterations as "removals" and "major changes." Information about these can be found in various official sources, including dedicated blog post series (like "Kubernetes Removal and Major Changes" and "Sneak Peek"), release announcement blog posts, the comprehensive change log, and the critical urgent upgrade notes. A key observation made by Mudrinić is that users often become aware of removals only when they upgrade and encounter issues. This highlights a crucial gap in user awareness regarding the deprecation cycle – a strict, multi-stage process where features are first marked for removal and only later, after a significant grace period and often with an alternative available, are they actually removed.

The necessity of breaking changes in Kubernetes stems from both philosophical and technical considerations. Philosophically, it's impossible for a project of Kubernetes' scale and ambition to evolve without making difficult choices. The primary driver is project healthiness and the well-being of its maintainers. Many Kubernetes maintainers are volunteers, dedicating their free time to the project. Asking them to perpetually maintain multiple, often redundant, implementations of the same functionality is unsustainable. A cleaner codebase, free from legacy cruft, also reduces the surface area for bugs and potential vulnerabilities, contributing to overall project stability. Technically, Kubernetes is a highly complex, almost microservices-like system, where different components are versioned independently. While core components remain stable, certain aspects, like CLI flags or configuration properties, or especially APIs, tend to evolve more frequently due to easier versioning mechanisms.

A recurring question is why Kubernetes hasn't pursued a "Kubernetes v2.0" to manage breaking changes. Mudrinić explained that such a drastic step would introduce immense confusion for users, break countless tools, and invalidate existing documentation and training materials. Crucially, the fundamental architecture and usage patterns (e.g., kubectl) of Kubernetes have remained remarkably consistent over the years. Existing mechanisms like API versioning (e.g., v1beta3 to v1) and the migration of CLI flags to configuration files effectively manage evolution without necessitating a complete system reset. A "v2.0" would only be considered for truly foundational shifts, such as the complete removal of the API server or a radical change in interaction paradigms (e.g., no longer accepting YAML).

Key Findings

▶ Watch: Where to find Kubernetes removals and major changes (3:10)

The talk revealed several critical findings that shape how users and contributors should approach Kubernetes:

1. Structured Deprecation for Core Features: Kubernetes adheres to a robust and lengthy deprecation policy for its core APIs, flags, features, and metrics. General Availability (GA) features, for instance, are deprecated for an average of 12 months or two releases (whichever is longer) before removal, providing ample time for users to adapt. This policy is a cornerstone of Kubernetes' commitment to stability and user predictability.
2. Mitigation-First Approach: The vast majority of breaking changes in Kubernetes are designed with a mitigation path. This means that while user action (e.g., updating manifests, changing configuration) is required, the underlying functionality or an equivalent alternative remains available, preventing a complete loss of capability. This is a deliberate strategy to balance innovation with operational continuity.
3. Infrastructure is an Exception – No Guarantees: A standout finding is the critical distinction between core Kubernetes features and its supporting infrastructure (e.g., container image registries like registry.k8s.io, binary download sites like dl.k8s.io, and package repositories). Due to reliance on volunteer efforts and donated cloud resources, these infrastructure components offer no guarantees regarding stability or longevity. Changes to infrastructure can occur with short notice or no notice at all, posing a significant risk to un-mirrored deployments.
4. Proactive Engagement is Paramount: The burden of navigating these changes largely falls on the user. Staying informed through official communication channels (change logs, mailing lists, social media) and proactively testing alternatives during deprecation periods are not merely suggestions but necessities for maintaining stable Kubernetes operations.
5. KEPs Drive Evolution: Every significant change in Kubernetes originates from a Kubernetes Enhancement Proposal (KEP). This rigorous, community-driven process, including detailed proposals, alternative considerations, and a Production Readiness Review (PRR), ensures that changes are thoroughly vetted, necessary, and safe for production environments.

Technical Deep Dive

▶ Watch: Understanding the Kubernetes deprecation cycle (4:45)

Kubernetes breaking changes manifest in several common forms, each with its own mitigation strategy. Mudrinić detailed three primary types of removals:

1. API Version Removals: This is perhaps the most common and impactful type. When an API version is deprecated (e.g., v1beta3), a newer, stable version (e.g., v1) is typically available. Kubernetes has an internal mechanism for converting objects from deprecated API versions to the stable ones upon creation or modification. While users might apply a manifest using an older API, Kubernetes stores and retrieves it using the new, stable version. Users are expected to update their manifests to use the current API version.
2. CLI Flag and Environment Variable Removals: Many command-line interface (CLI) flags or environment variables for core components like kubelet or kube-proxy have been deprecated and removed. These are almost universally replaced by structured configuration files. This shift promotes consistency, auditability, and easier management of complex component settings.
3. Built-in Feature Removals: In some cases, features initially built into Kubernetes are externalized. A prominent example is the Docker shim, which allowed kubelet to communicate directly with Docker. This was deprecated and eventually removed in Kubernetes v1.24, with users expected to transition to Container Runtime Interface (CRI)-compliant runtimes like CRI-O or containerd. Similarly, certain cloud-specific functionalities were moved out of the core into external Cloud Controller Managers (CCMs).

The lifecycle of any significant change in Kubernetes, including breaking changes, is governed by the Kubernetes Enhancement Proposal (KEP) process. A KEP is a detailed document hosted in the kubernetes/enhancements GitHub repository. It outlines the motivation, proposed solution, considered alternatives, implementation plan, test plan, and critically, a Production Readiness Review (PRR). This ensures that any new feature or significant modification is viable, stable, and production-ready. The KEP undergoes review by maintainers and a specialized group experienced in production operations, ensuring a high bar for quality and necessity.

The Kubernetes release cycle, typically spanning 13-14 weeks, incorporates several freezes to manage the flow of changes:

• Production Readiness Freeze (PRF): Approximately 11 weeks before a release, all KEPs must have a positive PRR rating.
• Enhancements Freeze: Around 10 weeks before a release, features must be implementable and approved to be considered for that release.
• Code and Test Freeze: Approximately 5 weeks before a release, all code must be merged, and tests completed.

Key policies govern these changes:

• Kubernetes Deprecation Policy: This comprehensive policy dictates the minimum deprecation periods:
• General Availability (GA) Features: 12 months or 2 releases, whichever is higher.
• Beta Features: 3 months or 1 release.
• Alpha Features: Can be removed at any time.

This policy covers APIs, CLI flags, features/behaviors, and metrics.

• Version Skew Policy: This policy defines the permissible version differences between Kubernetes components:
• kube-controller-manager, kube-scheduler, cloud-controller-manager: Must be at the same minor version as kube-apiserver or one minor version older.
• kubectl: Can be one minor version newer, older, or the same as kube-apiserver.
• kubelet, kube-proxy: Can be up to three minor versions older than kube-apiserver. This flexibility is crucial for large clusters and rolling upgrades.

A critical exception to these policies lies in Kubernetes infrastructure. Components like registry.k8s.io (for container images), dl.k8s.io (for binaries), and packages.k8s.io (for OS packages) offer no guarantees of stability or long-term availability. This is due to their reliance on volunteer maintenance and donated cloud credits. Past examples include the freezing of k8s.gcr.io (requiring a switch to registry.k8s.io), the adoption of Fastly CDN for dl.k8s.io to reduce costs, and the deprecation of legacy package repositories. These infrastructure changes can occur with little to no notice, posing a significant operational risk if not proactively managed by users.

Demo / Proof of Concept

▶ Watch: Philosophical reasons: project health and maintainer burden (6:07)

Marko Mudrinić provided a concise yet effective live demonstration of how Kubernetes handles API version conversions, specifically focusing on the deprecation of an API version.

The demonstration used a Kubernetes cluster running v1.31.2. He applied a FlowSchema resource manifest, intentionally using the deprecated v1beta3 API version for flowcontrol.apiserver.k8s.io. Upon executing kubectl apply -f flowschema-v1beta3.yaml, the command successfully created the resource but immediately issued a warning message, stating that v1beta3 was deprecated and would be unavailable in future versions (e.g., 1.32+), advising the use of v1 instead.

Crucially, when he then retrieved the created FlowSchema object using kubectl get flowschema <name> -o yaml, the output showed that Kubernetes had internally converted and stored the object using the v1 API version, not the v1beta3 that was supplied in the original manifest. This illustrated Kubernetes' automated conversion mechanism, where deprecated API objects are silently upgraded to their stable counterparts.

To further demonstrate control over API versions, Mudrinić showed how to explicitly request an object in a specific API version, even a deprecated one. Using kubectl get flowschema <name> -o yaml --api-version=flowcontrol.apiserver.k8s.io/v1beta3, he was able to retrieve the object formatted as v1beta3, again accompanied by the deprecation warning. This feature is useful for users who might need to compare manifests or understand how their objects would appear in older API versions, but it reinforces the expectation that v1 is the current and preferred version.

The demo effectively highlighted two key aspects: Kubernetes' internal handling of API version deprecation to ensure forward compatibility, and the tools available to users to inspect and manage these transitions.

Defensive Implications

▶ Watch: Technical reasons: complexity and component versioning (8:05)

Navigating the evolving landscape of Kubernetes requires a proactive and informed defensive strategy. Mudrinić outlined several key actions defenders should take:

1. Stay on Top of News and Development:

• Follow the Change Log: This is the most critical resource for identifying upcoming deprecations and breaking changes. Regular review is essential.
• Subscribe to Kubernetes Announce Mailing List: This low-traffic, high-importance list (around 5,000 subscribers) is used for critical information like CVEs, new releases, and short-notice breaking changes, especially those related to infrastructure.
• Monitor "Last Week in Kubernetes Development": While primarily for contributors, this weekly mailing list offers deep insights into ongoing development and proposed changes.
• Follow Official Social Media: Kubernetes maintains profiles on LinkedIn, Twitter, and Blue Sky, which often share important announcements.

1. Proactively Address Deprecations:

• Test Alternatives During Deprecation Periods: Do not wait until a feature is removed. Utilize the generous deprecation window (e.g., 12 months for GA APIs) to test and migrate to the recommended alternatives. Examples include migrating from Docker shim to CRI-O or containerd, and switching to external Cloud Controller Managers (CCMs).
• Provide Constructive Feedback: Engage with the Kubernetes community (via GitHub or Slack) on proposed alternatives. Focus feedback on the functionality, migration challenges, and real-world impact of the alternative, rather than requesting the reversal of a removal decision, which is highly unlikely.

1. Mirror All Kubernetes Infrastructure:

• This is paramount. Given that Kubernetes infrastructure (container image registries like registry.k8s.io, binary download sites like dl.k8s.io, and package repositories like packages.k8s.io) offers no guarantees and can change abruptly, organizations must mirror these artifacts locally or in private repositories.
• This includes container images (e.g., kube-apiserver, kube-controller-manager, kube-scheduler, etcd, coredns), binaries, and OS packages. Instructions for mirroring images from registry.k8s.io are available.
• Mirroring ensures operational continuity, protecting clusters from upstream outages, unexpected changes, or regional access issues. It also indirectly helps the Kubernetes project by reducing its cloud hosting costs.

1. Understand Policy Nuances:

• Be aware of the Deprecation Policy timelines (12 months/2 releases for GA, 3 months/1 release for Beta, immediate for Alpha) to prioritize migration efforts.
• Familiarize yourself with the Version Skew Policy to plan safe upgrade paths for kubelet, kube-proxy, kubectl, and other control plane components relative to the kube-apiserver.

By implementing these defensive strategies, organizations can significantly reduce the risk and operational overhead associated with Kubernetes' inevitable evolution, ensuring stable and secure cluster operations.

Key Takeaways

• Breaking Changes are Inevitable but Managed: Kubernetes embraces breaking changes as necessary for project health and evolution, but most core feature removals come with extensive notice and mitigation paths.
• Deprecation Policy Provides Ample Warning: For General Availability (GA) features, there's typically a 12-month or two-release deprecation period, giving users significant time to adapt. Beta features have shorter cycles, and Alpha features can be removed at any time.
• Kubernetes Infrastructure Has No Guarantees: Critical infrastructure components like container image registries (registry.k8s.io) and package repositories (packages.k8s.io) are maintained by volunteers and rely on donations, meaning they can change or become unavailable with little to no notice.
• Mirror All the Things: To ensure operational resilience and protect against infrastructure changes, it is crucial for users to mirror all Kubernetes-related container images, binaries, and packages to their own private repositories.
• Proactive Information Gathering is Key: Staying informed through official channels (change logs, Kubernetes announce mailing list, social media) and actively monitoring deprecation notices is vital for smooth upgrades and avoiding unexpected issues.
• Engage Early and Constructively: Provide feedback on proposed alternatives during deprecation periods via GitHub or Slack, focusing on practical implementation and migration challenges rather than requesting reversals of well-reasoned removal decisions.

About the Speaker(s)

Marko Mudrinić is a Senior Software Engineer at Kubermatic, a company specializing in Kubernetes automation and management. Beyond his corporate role, Mudrinić is deeply embedded in the Kubernetes open-source community, holding significant leadership positions. He serves as the SIG K-infra Tech Lead, overseeing the infrastructure that supports the Kubernetes project itself, and is also a sub-project lead for Kubernetes Release Engineering, playing a crucial role in the planning and execution of Kubernetes releases. In recognition of his contributions and expertise, Marko Mudrinić is also a CNCF Ambassador, advocating for cloud-native technologies and fostering community engagement. His multifaceted involvement provides him with a unique perspective on both the technical and community aspects of Kubernetes development and operations.