What's New With Kubectl and Kustomiz... Eddie Zaneski, Maciej Szulik, Marly Salazar & Yugo Kobayashi

What's New With Kubectl and Kustomize: Community-Driven Enhancements for the Kubernetes CLI

Speakers: Eddie Zaneski, Maciej Szulik, Marly Salazar, Yugo Kobayashi

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=KQBz7nwWxUE

Overview

This talk, presented by the Special Interest Group (SIG) CLI leads Eddie Zaneski, Maciej Szulik, Marly Salazar, and Kustomize maintainer Yugo Kobayashi, offers a comprehensive update on the latest developments in kubectl and Kustomize, the indispensable command-line tools for interacting with Kubernetes clusters. Beyond just detailing new features, the speakers emphasized the critical role of community involvement in shaping the future of these widely used tools. The session highlighted significant improvements in user experience, security, and configurability, addressing long-standing pain points and introducing innovative ways for users to tailor their Kubernetes workflows.

The presentation underscored the challenges inherent in maintaining core Kubernetes components, particularly the strict adherence to backward compatibility and the volunteer-driven nature of open-source development. From preventing accidental cluster deletions to enhancing the security of file transfers and simplifying configuration management, the SIG CLI team is focused on making kubectl and Kustomize more robust, intuitive, and secure. This talk is essential for any Kubernetes user, administrator, or developer looking to leverage the latest CLI capabilities and understand the roadmap for these foundational tools.

Background

▶ Watch: Introduction and SIG CLI's mission (0:00)

The SIG CLI (Special Interest Group for the Command Line Interface) is a vital part of the Kubernetes project, responsible for maintaining and evolving the core command-line tools that users interact with daily, most notably kubectl and Kustomize. As explained by the speakers, SIGs are open groups within Kubernetes where contributors volunteer their time to own specific parts of the codebase. This volunteer-driven model, while powerful, also presents challenges, such as maintaining momentum and addressing a vast backlog of issues, especially when key contributors step away.

A recurring theme throughout the talk was the paramount importance of backward compatibility in Kubernetes. The project maintains a very strict contract to ensure that upgrades do not inadvertently break existing workflows, particularly automated CI/CD pipelines. This constraint often necessitates creative solutions for introducing new, safer behaviors without altering established defaults, which could otherwise "page a whole bunch of people" as Eddie Zaneski noted. This is why many new features are introduced as opt-in flags or configurable preferences rather than immediate default changes.

The development process for significant new features in Kubernetes typically follows KEPs (Kubernetes Enhancement Proposals), which outline the design, scope, and stages (Alpha, Beta, GA) of large-scale implementations. The talk referenced several KEPs that have driven the recent enhancements, reflecting a structured approach to evolving the CLI. Prior to these updates, users faced issues like the ease of accidentally wiping a cluster with kubectl delete --all, the limitations of kubectl create for custom resource definitions, and security vulnerabilities associated with kubectl copy due to its reliance on external binaries. These existing problems formed the impetus for many of the improvements discussed.

Key Findings

▶ Watch: Understanding Kubernetes Enhancement Proposals (KEPs) (2:00)

The talk unveiled several significant enhancements and strategic decisions impacting kubectl and Kustomize, demonstrating a clear focus on improving user experience, security, and configurability:

• Kustomize's Continued Integration: A key decision was made to not remove Kustomize from kubectl, despite earlier proposals. This was driven by a strong desire to support existing projects and prioritize backward compatibility, ensuring stability for current users.
• Enhanced kubectl debug with Custom Profiles: kubectl debug now supports custom profiles, allowing administrators to define specialized sets of permissions and rules for debugging sessions. This provides greater flexibility and security beyond the legacy tooling, which remains supported.
• Subplugins for kubectl create: To address the challenge of flag proliferation and limited resource creation options, kubectl create is being augmented with subplugins. This enables the community to define custom creation logic for specific resources, moving away from an ever-growing list of built-in flags.
• Subresource Support for kubectl get: The kubectl get command now supports subresources, providing more granular and readable table output for specific aspects of resources, such as the status or scale of a deployment. This feature is slated for GA (General Availability) in Kubernetes 1.33.
• Interactive Delete for kubectl delete (KEP 3895): A critical safety feature, the -i flag, introduces interactive delete confirmation for kubectl delete operations. This directly addresses the long-standing problem of accidental cluster wipes, offering users an opt-in prompt before destructive actions. It is alpha in Kubernetes 1.33.
• User Preferences via QRC File: A groundbreaking new feature is the introduction of a separate QRC file (e.g., .kube/qrc) for user-specific kubectl preferences. This file allows users to define aliases and, more importantly, override default kubectl behaviors, such as forcing delete to always be interactive or apply to always use server-side logic. This is also alpha in 1.33 and provides a powerful mechanism for enforcing safer defaults, especially in managed environments.
• Kustomize Release Automation and Helm Integration: Future plans for Kustomize include implementing release automation for its binary, enabling more frequent releases. Furthermore, significant work is underway to improve Helm support by importing Helm code directly rather than executing the Helm binary, which addresses security concerns and simplifies maintenance.

These findings collectively represent a substantial leap forward in the usability, safety, and extensibility of Kubernetes' primary command-line interface.

Technical Deep Dive

▶ Watch: Improving kubectl create with user-defined subplugins (3:15)

The technical discussions revolved around significant architectural and functional enhancements to kubectl and Kustomize, many of which are designed to address long-standing challenges related to security, usability, and maintainability.

Kustomize Evolution

Yugo Kobayashi detailed the strategic decision to retain Kustomize within kubectl, emphasizing the priority of supporting existing projects and maintaining backward compatibility. This decision underscores the widespread adoption of Kustomize for Kubernetes configuration management. Future plans for Kustomize focus on three key areas:

1. Plugin Support: The project aims to support plugins, akin to the current exec container KEP plugins, allowing for greater extensibility and customization of Kustomize's behavior.
2. Release Automation: Implementing release automation for the Kustomize binary will enable more frequent releases, allowing users to access new features and bug fixes faster.
3. Improved Helm Support: A critical security and maintenance enhancement involves fundamentally changing how Kustomize interacts with Helm. Currently, Kustomize executes the Helm binary on the user's system. This approach presents security risks and maintenance overhead. The plan is to move towards importing Helm code directly into Kustomize. This change, which is being coordinated with Helm maintainers for Helm V4 (as much of the Helm code is not currently exported as a library), will mitigate security concerns related to external binary execution and create a more robust, integrated experience.

kubectl Enhancements

The kubectl team introduced several features, many of which are being rolled out as KEPs (Kubernetes Enhancement Proposals), ensuring a structured and reviewed development process.

• Custom Profiles for kubectl debug: This feature extends kubectl debug to allow administrators to provide custom profiles for debugging sessions. Previously, kubectl debug had a fixed set of permissions and rules. Now, admins can pass in arbitrary patches to specialize debugging events, offering granular control over the debug container's configuration and permissions. This maintains backward compatibility by allowing legacy tooling to function while providing enhanced flexibility.
• Subplugins for kubectl create: Addressing the notorious "flag proliferation" problem, where kubectl create commands accumulate an unmanageable number of flags for various resource types, the team is introducing subplugins. These plugins allow the community to define custom creation logic for specific resources. This approach offloads the maintenance burden from the core kubectl team and empowers users to tailor kubectl create to their unique resource definitions and workflows, especially for Custom Resource Definitions (CRDs).
• Subresource Support: kubectl get now supports displaying specific subresources for various Kubernetes objects. For instance, instead of just getting generic deployment information, users can now directly query and display the status or scale subresources of a deployment in a clean, table-formatted output. This feature significantly improves the clarity and utility of kubectl get for specific operational tasks and is scheduled to be GA in Kubernetes 1.33.
• Interactive Delete (KEP 3895): This feature introduces the -i flag to kubectl delete, enabling interactive confirmation before executing deletion commands. The backstory highlights a critical design challenge: simply changing the default behavior would break countless CI/CD pipelines that rely on non-interactive deletion. Therefore, an opt-in -i flag was developed, providing a safety net for manual operations without disrupting automation. This feature is alpha in Kubernetes 1.33, meaning it requires an explicit environment variable to be enabled.
• User Preferences (QRC File): To provide a persistent and configurable way to set kubectl defaults, a new concept of user preferences stored in a .kube/qrc file (or passed via a flag) has been introduced. This file allows users to:
• Define aliases: For example, get store-pods could be aliased to kubectl get pods -n mystore --selector app=store.
• Set overrides: Crucially, this allows users to force specific kubectl behaviors by default. Examples include always using server-side apply (apply.defaulting=serverside) or making delete always interactive (delete.interactive=true). This feature, also alpha in 1.33, offers a powerful mechanism for administrators to provision developer machines with safer, standardized kubectl behaviors. The API shape for the QRC file was meticulously reviewed by Jordan, ensuring its robustness.

Future Work and Challenges

The SIG CLI team outlined several areas for future development, many of which tackle complex, long-standing issues:

• Multiple Kubeconfigs: The current context mechanism helps manage different configurations within a single kubeconfig file, but the team is exploring how to better support scenarios involving entirely separate kubeconfig files, focusing on user experience.
• New Server-Side Apply Command: Recognizing the benefits of server-side apply (e.g., preventing configuration drift, better merge semantics), the team wants to introduce a new kubectl command that defaults to server-side apply. The primary challenge is finding a suitable name that avoids generic terms like "apply v2" or awkward ones like "actuate." This change, like interactive delete, cannot simply alter the default kubectl apply behavior due to backward compatibility.
• JSONPath Improvements: The existing kubectl JSONPath implementation is a limited subset of actual JSONPath, lacking many expected features (e.g., the length function). The team is considering whether to enhance the current implementation or adopt alternatives like CEL (Common Expression Language) for more powerful and spec-compliant querying.
• CRI-Native Container Copy: The kubectl copy command has been a source of significant security vulnerabilities, with the speakers recalling at least three CVEs related to path traversal issues. Its current implementation relies on the tar binary being present both locally and within the target container, and its functionality has been severely limited (e.g., to single-file copies) to mitigate risks. The proposed solution is to push the file copy API directly into the CRI (Container Runtime Interface), allowing container runtimes like Docker, containerd, and CRI-O to handle the operation securely at the node and kubelet level. This would bypass the need for external binaries and significantly reduce the attack surface.
• Flag Reduction: kubectl suffers from an overwhelming number of flags, to the point where the help text itself had to be rewritten to be readable. The team is committed to avoiding the introduction of new flags where possible and finding alternative extensibility mechanisms like subplugins.
• Improved Kustomize Pruning: The talk briefly touched upon an ongoing effort to develop a more reliable pruning mechanism for Kustomize, which is currently in alpha. This aims to improve how resources are applied and subsequently cleaned up, with a goal to move it to default in Kubernetes 1.34.

These technical deep dives reveal a SIG CLI team that is not only responsive to immediate user needs but also strategically planning for the long-term security, maintainability, and extensibility of Kubernetes' core CLI tools.

Demo / Proof of Concept

▶ Watch: Enhanced subresource support for kubectl table output (4:00)

Maciej Szulik provided a concise and effective demonstration of the new User Preferences feature, showcasing how the .kube/qrc file can transform the kubectl experience.

The demo began by displaying a sample .kube/qrc file, illustrating its structure. The file can be explicitly named qrc and placed in the user's .kube directory, or its path can be passed via a flag. Maciej highlighted two primary functionalities:

1. Aliases: The qrc file can define aliases for kubectl commands. An example shown was an alias for get store-pods, which internally expands to kubectl get pods -n <namespace> --selector app=store. The demo illustrated how arguments can be appended either before or after the aliased command's arguments, offering flexibility in command construction. Maciej successfully executed kubectl get store-pots (a typo in the demo, but the intent was clear) and kubectl get store-deployments, demonstrating how these aliases simplify complex or frequently used commands.
2. Overrides: This is where the .kube/qrc file truly shines in terms of enforcing safer defaults. Maciej demonstrated two critical overrides:

• Server-Side Apply Defaulting: An entry like apply.defaulting=serverside in the qrc file automatically forces all kubectl apply commands to use server-side apply. The output clearly showed "pod/nginx configured (server-side apply)" instead of the standard "pod/nginx configured," confirming the override was active without requiring the user to explicitly add --server-side to every command.
• Interactive Delete: The most impactful override, delete.interactive=true, automatically adds the -i flag to all kubectl delete commands. When Maciej ran kubectl delete pod/nginx, the terminal immediately prompted, "Are you sure you want to delete pod/nginx?" This confirmed the interactive confirmation was enforced by default, preventing accidental deletions.

Maciej also took a moment to issue a strong warning against the deprecated and dangerous kubectl delete all --all command. He clarified that all as an alias for resources does not mean all resources in the cluster, and the --all flag (which targets all resources of the specified type) combined with the all alias can lead to unintentional, catastrophic cluster wipes without any prompt. The interactive delete override via the qrc file offers a crucial safeguard against such mistakes.

The demo effectively illustrated the power of user preferences in streamlining workflows and enhancing safety, particularly for administrators who wish to provision developer environments with standardized and secure kubectl defaults.

Defensive Implications

▶ Watch: Introducing user preferences via the QRC file (8:10)

The advancements discussed in the SIG CLI talk carry significant defensive implications for Kubernetes administrators and security teams, offering new tools and strategies to enhance cluster security and operational resilience.

1. Preventing Accidental Deletions with Interactive Delete and QRC: The introduction of the interactive delete (-i flag) and its enforcement through the QRC user preferences file is a critical security enhancement. Accidental deletion of namespaces or entire clusters is a well-documented risk. By allowing administrators to mandate interactive confirmation for kubectl delete operations via the QRC file, especially during developer laptop provisioning, organizations can drastically reduce the likelihood of human error leading to catastrophic data loss or service outages. This acts as a vital last line of defense for manual operations, without impacting automated CI/CD pipelines that do not use the -i flag.
2. Enforcing Server-Side Apply by Default: The capability to set apply.defaulting=serverside in the QRC file encourages, or even enforces, the use of server-side apply. Server-side apply is a more robust mechanism for managing Kubernetes object configurations as it tracks field ownership, prevents configuration drift, and handles merge conflicts more gracefully than client-side apply. By making this the default, defenders can ensure more consistent and predictable application of configurations, reducing the attack surface created by misconfigurations or unexpected state changes.
3. Enhanced Security for kubectl copy (CRI-Native): The planned move to a CRI-native container copy mechanism is a major defensive improvement. The current kubectl copy has been a source of multiple CVEs due to its reliance on external tar binaries and susceptibility to path traversal vulnerabilities. By pushing this functionality into the Container Runtime Interface (CRI) at the node/kubelet level, the operation can be performed in a more controlled and secure environment, significantly reducing the risk of malicious file transfers or container escapes. This directly addresses a critical attack vector that has plagued kubectl for years.
4. Secure Kustomize-Helm Integration: The plan to integrate Helm by importing its code as a library rather than executing an external binary improves the supply chain security of Kustomize. Relying on external binaries introduces risks if the binary itself is compromised or behaves unexpectedly. By tightly integrating Helm's functionality, Kustomize can ensure a more secure and controlled environment for generating Kubernetes manifests from Helm charts, reducing the potential for injection attacks or unintended side effects.
5. Custom Profiles for kubectl debug: Providing administrators with the ability to define custom profiles for kubectl debug allows for more granular control over the permissions and capabilities granted to debugging containers. This can help enforce the principle of least privilege, ensuring that debug sessions have only the necessary access, thereby minimizing the potential impact of a compromised debug session.
6. Subplugins for kubectl create: By enabling community-driven subplugins for kubectl create, the core kubectl binary becomes less bloated and more modular. This reduces the attack surface by limiting the amount of code directly maintained within kubectl for specific resource creation logic. If a vulnerability were found, it might be isolated to a specific plugin rather than the core tool.

In summary, the SIG CLI's work provides administrators with powerful new controls to harden their Kubernetes environments, ranging from preventing human error in daily operations to mitigating long-standing security vulnerabilities in core functionalities. The emphasis on user preferences, secure integrations, and controlled extensibility empowers defensive strategies across the Kubernetes ecosystem.

Key Takeaways

• User Preferences (QRC File) Revolutionize kubectl Configuration: The new .kube/qrc file allows users and administrators to define aliases and, crucially, enforce default behaviors like interactive delete and server-side apply, significantly enhancing safety and workflow consistency.
• Critical Safety Features Address Long-Standing Risks: Interactive delete (-i flag) for kubectl delete directly tackles the problem of accidental cluster wipes, providing an opt-in safeguard that can be centrally managed.
• Major Security Improvements Planned for kubectl copy and Kustomize's Helm Integration: The shift towards CRI-native kubectl copy aims to mitigate severe path traversal CVEs, while importing Helm code into Kustomize will reduce supply chain risks associated with external binary execution.
• Kustomize Remains a Core Component with Future Enhancements: Despite prior discussions, Kustomize will remain integrated with kubectl, with ongoing plans for plugin support, release automation, and a more secure Helm integration.
• Community Contribution is Essential for Tool Evolution: The SIG CLI operates on volunteer contributions, making community involvement critical for addressing the vast backlog, developing new features, and maintaining the stability of these foundational Kubernetes tools.
• kubectl is Becoming More Flexible and Specific: Enhancements like custom profiles for kubectl debug and subresource support for kubectl get provide greater control and more precise information, while subplugins for kubectl create offer a scalable solution for managing diverse resource types.

About the Speaker(s)

The talk was presented by a team of dedicated maintainers and leads from the Kubernetes SIG CLI, the Special Interest Group responsible for the Kubernetes Command Line Interface tooling.

• Eddie Zaneski is a SIG CLI lead, actively involved in guiding the direction and development of kubectl and related tools. He emphasized the importance of community contributions and the challenges of maintaining backward compatibility.
• Maciej Szulik is also a SIG CLI lead, who demonstrated the new user preferences (QRC file) and provided insights into the technical implementation and future roadmap of various kubectl features.
• Marly Salazar serves as a SIG CLI lead, contributing to the overall strategic planning and enhancement proposals for the CLI tools.
• Yugo Kobayashi is a Kustomize maintainer and mentor. He provided key updates on Kustomize's status, including the decision to keep it integrated with kubectl, its future plans for plugins, release automation, and the critical improvements planned for Helm integration.

Together, these individuals represent the core team driving the evolution of the essential command-line tools that underpin the Kubernetes user experience. They highlighted the volunteer nature of their work and the ongoing need for community involvement to sustain and advance the project.