Strengthening Auth in Kubernetes: Image Pulling, DRA Admin Acces... Rita Zhang & Stanislav Láznička

Strengthening Auth in Kubernetes: Image Pulling, DRA Admin Access, and Beyond

Speakers: Rita Zhang, Engineer at Microsoft & SIG Auth Co-chair; Stanislav Láznička, Engineer at Microsoft & SIG Auth Contributor

Conference: KubeCon EU

YouTube: <https://www.youtube.com/watch?v=rVz-vIFGT4k>

Overview

This talk, presented by Rita Zhang and Stanislav Láznička of Microsoft and SIG Auth co-chair and contributor respectively, provides a comprehensive update on the latest advancements and future directions in Kubernetes authentication and authorization. The session highlights key features graduating to GA (General Availability) in Kubernetes v1.33, alongside promising alpha and in-flight features planned for future releases like v1.34. The core theme revolves around enhancing the security posture of Kubernetes clusters by adopting the principle of least privilege, reducing reliance on long-lived secrets, and improving identity verification mechanisms across various components.

The speakers delve into critical areas such as securing container image pulling, refining administrative access for dynamic resource allocation, and streamlining certificate management within the cluster. They also touch upon foundational improvements to service account tokens and Kubelet API authorization, underscoring the continuous effort by the Kubernetes community to harden the platform against evolving threats. Through detailed explanations and practical demonstrations, Zhang and Láznička illustrate how these enhancements empower cluster administrators to build more robust, auditable, and secure Kubernetes environments.

The importance of this presentation for the Kubernetes community cannot be overstated. As Kubernetes adoption grows, so does the complexity of managing secure workloads and infrastructure. The features discussed directly address common security pain points, such as unauthorized image access, overly broad permissions, and cumbersome certificate rotations. By providing granular control over critical operations and leveraging stronger identity primitives, SIG Auth is paving the way for a more secure-by-default Kubernetes experience, ultimately benefiting developers, operators, and security professionals alike.

Background

▶ Watch: Introduction and SIG Auth updates overview (0:00)

Historically, Kubernetes has provided a robust, yet often complex, set of authentication and authorization primitives. However, certain aspects presented challenges or opportunities for improvement from a security perspective. For instance, managing trusted certificates across a cluster could be cumbersome, often involving manual configuration of numerous ConfigMaps and complex rotation strategies. Similarly, the default behavior for container image pulling, particularly when using private registries, presented a notable security gap. Once an image was pulled to a node by a privileged pod, any other pod scheduled on that node could potentially access and run that image without requiring its own authentication to the registry, undermining the principle of least privilege.

Authorization for critical components like the Kubelet API also historically lacked granularity, often forcing administrators to grant broad proxy permissions where only specific endpoint access was needed. This created an elevated risk profile for monitoring or logging agents that only required limited Kubelet interaction. Furthermore, the reliance on long-lived secrets for various operations, including image pulling, has always been a security concern, increasing the attack surface and making key rotation a difficult operational burden.

SIG Auth (Special Interest Group for Authentication) in the Kubernetes community is dedicated to addressing these challenges. Their work focuses on developing and refining features that enhance identity verification, enforce granular access controls, and improve the overall security architecture of Kubernetes. The features presented in this talk represent a significant stride in mitigating these long-standing issues, moving Kubernetes towards a more secure, identity-driven, and least-privileged operational model. This continuous effort is crucial for adapting Kubernetes to increasingly stringent security requirements and complex multi-tenant environments.

Key Findings

▶ Watch: Cluster Trust Bundles for easier trust management (2:00)

The talk presented a series of significant updates and upcoming features from SIG Auth, categorized by their release status and impact on Kubernetes security:

Graduated Features (GA in v1.33):

• Bound Service Account Token Improvements: This feature reaches GA in v1.33, providing a more robust chain of identity verification for service account tokens by embedding additional claims like node name and JTI (JWT ID), thus preventing replay attacks.
• Cluster Trust Bundles: Also graduating to GA, this simplifies the installation and maintenance of additional trusted signers within a cluster, replacing manual ConfigMap management with an API object approach.
• Fine-grained Kubelet API Authorization: Moving to beta in v1.33, this allows administrators to grant granular permissions for specific Kubelet API endpoints (e.g., configz, pprof) without resorting to broader proxy permissions, adhering to the principle of least privilege.
• UID Enhancements for Authenticators: Enhancements for UIDs are now honored in the Client Certificate Authenticator and the Request Header Authenticator (graduating to beta), providing more consistent user identification across authentication methods.

Upcoming and Alpha Features (v1.33 Alpha and beyond):

• Image Pull Credential (Alpha in v1.33): This is a critical security improvement that enforces re-authentication to the image registry if a pod attempts to use an image previously pulled to a node by different credentials. It aims to improve the default security stance of Kubernetes.
• Service Account Image Pull Credential: This feature, part of a broader effort to reduce reliance on long-lived secrets, enables the Kubelet to request short-lived tokens tailored to specific audiences for image pulling, using projected service account tokens.
• DRA Admin Access (Alpha in v1.33): For Dynamic Resource Allocation (DRA), this feature introduces mechanisms to allow administrators to manage specialized hardware or resources without impacting normal users' ability to allocate those resources, by using specific labels on namespaces.
• External Signing for Service Account Tokens (Alpha in v1.33): This allows the Kubernetes API server to offload token signing to external systems like KMS (Key Management System) via gRPC calls, simplifying key rotation, verification, and auditing.
• P-certificates for Service Accounts (In-flight): This aims to simplify the minting of client certificates for service account identities, introducing new APIs and integrating with node authorization.

In-flight Features (Planned for v1.34 and future releases):

• Harden Kubelet Server Validation: Adds an additional check for the kube-apiserver to ensure the Kubelet's hostname matches expectations during connection.
• Extension of Node Restriction to Service Accounts: Limits the scope of service accounts, particularly for daemonsets, to only access the specific node object they are associated with.
• PSA Restrictions for Probes Host Fields: A new Pod Security Admission (PSA) control to mitigate Server-Side Request Forgery (SSRF) vulnerabilities by restricting the doHostFile setting in probe handler and lifecycle handler configurations.

These findings collectively demonstrate a strong commitment from SIG Auth to continuously elevate the security posture of Kubernetes, addressing both long-standing issues and emerging threats with robust, technically sound solutions.

Technical Deep Dive

▶ Watch: Fine-grained Kubelet API authorization for least privilege (3:30)

The features discussed represent significant architectural and functional enhancements, each designed to bolster specific aspects of Kubernetes security.

Bound Service Account Token Improvements (GA in v1.33):

Historically, verifying a service account token's association with a specific pod and node was complex. This feature addresses that by embedding crucial information directly into the JSON Web Token (JWT). Specifically, it includes the node name where the pod is scheduled and a JTI (JWT ID) claim. The node name allows for direct verification that the token originated from a pod on a specific node, enhancing the chain of identity verification. The JTI prevents replay attacks by ensuring each token is unique and can only be used once or within a specific context, making it much harder for attackers to reuse stolen tokens. This moves away from the previous method of cross-referencing private claims with pod objects to determine node association. Mo, James, and Jordan were key contributors to this enhancement.

Cluster Trust Bundles (GA in v1.33):

Managing trusted Certificate Authorities (CAs) in Kubernetes has often involved distributing ConfigMaps containing CA certificates, which could become cumbersome, especially during rotation. Cluster Trust Bundles introduce a dedicated API object for defining trusted signers. Instead of juggling ConfigMaps, administrators can define their trusted CAs as ClusterTrustBundle objects. Pods can then mount these bundles as projected volumes, specifying the desired signer name and optionally using label selectors to include multiple CAs (e.g., for rotation). This simplifies the process of installing, maintaining, and rotating trust within the cluster, making it easier to secure internal and external communications. Stanislav Láznička was instrumental in moving this feature to beta.

Fine-grained Kubelet API Authorization (Beta in v1.33):

Led by Vineiac, this feature enables administrators to enforce the principle of least privilege more effectively for Kubelet API access. Previously, components like logging or monitoring agents often required broad proxy permissions to access specific Kubelet endpoints such as /configz or /pprof. This new capability allows for authorization policies that grant access only to the necessary Kubelet API endpoints, significantly reducing the potential attack surface. This ensures that a compromised monitoring agent, for example, cannot exploit broader Kubelet permissions to perform actions beyond its intended scope.

UID Enhancements for Authenticators (v1.33):

User IDs (UIDs) are crucial for external systems and for consistent identity management. This enhancement integrates UIDs into two key authenticators. For the Client Certificate Authenticator, the kube-apiserver now honors UIDs found in a specific Relative Distinguished Name (RDN) field within the client certificate's subject, identified by the OID 1.3.6.1.4.1.34380.2.2. For the Request Header Authenticator, which graduates from alpha to beta, the kube-apiserver will now always include the UID in the X-Remote-UID header by default. While consuming these headers still requires configuration, this provides a standardized way to propagate and utilize UIDs across different authentication contexts, improving traceability and integration with external identity systems.

Image Pull Credential (Alpha in v1.33):

This feature, a major security improvement led by Stanislav Láznička, addresses a critical vulnerability: once an image from a private registry is successfully pulled to a node, any other pod on that node could previously use it without re-authentication, even if it lacked the original credentials. The new mechanism records the credentials used for each successful image pull on the node. When a new pod attempts to access an image already present on the node, the Kubelet checks if the pod's presented credentials match those previously used. If they don't, the Kubelet forces a re-authentication attempt against the registry. This prevents unauthorized reuse of cached images and eliminates the "weird behavior" of IfNotPresent and Never image pull policies potentially allowing access to images without proper authorization. It also removes the container registry from the critical path for subsequent pulls by authorized pods, improving resilience.

Service Account Image Pull Credential:

Anish contributed to this feature, which further refines image pulling security by reducing reliance on long-lived secrets. Instead of static imagePullSecrets stored in the Kubernetes API, this feature allows the Kubelet to request short-lived tokens for image pulling. It leverages projected service account tokens for Kubelet credential providers, enabling dynamic configuration of service account names and audiences for these token requests. This means pods can use their own identity to pull images, and the credentials are short-lived and specific to the task, greatly enhancing security and simplifying secret management and rotation.

DRA Admin Access (Alpha in v1.33):

Developed in collaboration with SIG Node, Dynamic Resource Allocation (DRA) introduces new APIs for dynamic requesting and sharing of specialized hardware like GPUs. The DRA Admin Access feature ensures that administrative tasks requiring exclusive access to these resources do not inadvertently starve normal user workloads. It achieves this by allowing specific resource claims and resource claim templates to be created with elevated privileges. These admin-privileged claims can only be used within namespaces explicitly labeled with resource.k8s.io/admin-access: "true", preventing non-admins from misusing this capability to monopolize resources. This is crucial for maintaining fairness and operational stability in multi-tenant GPU or specialized hardware clusters.

External Signing for Service Account Tokens (Alpha in v1.33):

Jordan Samuel Harshel worked on this feature, which tackles the challenge of managing service account signing keys directly on the kube-apiserver. By allowing the kube-apiserver to make gRPC calls to external systems like KMS (Key Management System) for token signing, it offloads the responsibility of managing signing keys. This significantly simplifies key rotation, verification, and auditing processes, as these sensitive operations can now be handled by dedicated, highly secure external services, reducing the operational burden and risk associated with in-process key management.

P-certificates for Service Accounts (In-flight):

Tahir is leading this effort to simplify the minting of client certificates for service account identities. This involves introducing new APIs and integrating with node authorization. The goal is to make it easy to obtain client certificates for service accounts, which can then be used for mTLS (mutual Transport Layer Security) or other client-side authentication scenarios. The identity for a service account in this context is represented by a structure that includes a specific OID for the CNCF group: 1.3.6.1.4.1.34380.2.1, indicating its unique role.

Harden Kubelet Server Validation (In-flight):

This upcoming feature adds an extra layer of security when the kube-apiserver connects to the Kubelet. Beyond standard hostname validation, the API server will perform an additional check to ensure the Kubelet's reported hostname actually matches what the API server expects for that node. This prevents potential man-in-the-middle attacks or misconfigurations where a rogue Kubelet might impersonate a legitimate one.

Extension of Node Restriction to Service Accounts (In-flight):

This feature aims to tighten the scope of service accounts, particularly for workloads like daemonsets that label nodes. It restricts these service accounts to only access the specific node object they are associated with, preventing them from interacting with or gathering information about other nodes in the cluster. This enhances the node restriction admission controller by applying its principles more broadly to service accounts, further limiting lateral movement possibilities.

PSA Restrictions for Probes Host Fields (In-flight):

Sora is designing a new Pod Security Admission (PSA) control, targeted for alpha in v1.34, to mitigate SSRF (Server-Side Request Forgery) vulnerabilities. This control will restrict setting the doHostFile field within probe handler and lifecycle handler configurations. Improperly configured probes or lifecycle hooks that allow arbitrary host file access can be exploited to perform SSRF attacks, enabling attackers to make requests from the pod to internal services or metadata endpoints. This restriction will prevent such vectors, enhancing the security of workload definitions.

Demo / Proof of Concept

▶ Watch: UIDs in Authenticators: Client Cert and Request Header (4:15)

The talk included three practical demonstrations showcasing the functionality and impact of several new features.

DRA Admin Access Demo

The first demo illustrated the DRA Admin Access feature in a Kubernetes cluster with GPUs. The scenario involved a cluster where regular AI workloads were allocating all available GPUs using standard resource claims and resource claim templates. An administrative task, such as collecting health metrics from the GPUs, would typically fail to schedule because all resources were already reserved, resulting in an error.

To address this, the demo showed the following steps:

1. Initial State: Multiple worker nodes with GPUs, and regular AI workloads deployed, consuming all available GPU resources. An admin pod attempting to allocate resources fails due to unavailability.
2. Namespace Labeling: A new namespace was created and explicitly labeled with resource.k8s.io/admin-access: "true". This label signals to the Kubernetes API server that this namespace is designated for administrative tasks that may require elevated resource access.
3. Admin Pod Redeployment: The admin pod's YAML was updated to use a new resource claim template that specified the admin-access feature and was deployed into the newly labeled namespace.
4. Successful Allocation: With the resource.k8s.io/admin-access: "true" label, the admin pod successfully scheduled and ran, demonstrating that administrative tasks can now acquire necessary resources without being blocked by or preventing normal user workloads. This ensures that critical maintenance or monitoring operations can proceed even in highly utilized clusters.

Image Pull Credential Demo

This demo highlighted the security improvement brought by the Image Pull Credential feature, which prevents unauthorized reuse of container images cached on a node. The demo contrasted the "old behavior" (without the feature) with the "new behavior" (with the feature enabled).

1. Unauthorized Pull Attempt: Initially, a pod was created attempting to pull a private image without any credentials, which correctly failed on both old and new clusters.
2. Authorized Pull (Old vs. New): A pull secret was created, and a pod (pod-with-secret) was launched using this secret to pull the private image. This successfully pulled the image to the node.
3. Unauthorized Reuse Attempt: A different pod (broken-pod) was then created without the pull secret, attempting to use the same image now cached on the node.

• Old Behavior: On the cluster without the feature, the broken-pod successfully started because the image was already present on the node, and it didn't re-authenticate. This exposed the private image to an unauthorized pod.
• New Behavior: On the cluster with the Image Pull Credential feature enabled, the broken-pod failed to start. The Kubelet detected that the broken-pod did not present the credentials originally used to pull the image and forced a re-authentication, which failed. The logs explicitly showed that the image failed to pull because it was forced to re-pull from the registry.

1. pullPolicy: Never Scenario: The demo further showed that even with pullPolicy: Never, the new configuration still prevented the unauthorized pod from starting, reinforcing the security guarantee.
2. Multi-tenancy: The demo extended to a multi-tenancy scenario, creating pods in different namespaces. The results remained consistent: the old cluster allowed unauthorized reuse, while the new cluster enforced re-authentication, demonstrating the feature's effectiveness across namespaces.

Cluster Trust Bundles Demo

The final demo showcased the simplified management of trusted CAs using Cluster Trust Bundles.

1. Feature Gate Configuration: The demo began by configuring feature gates for ClusterTrustBundle on the API server, controller manager, and Kubelet. This enabled the necessary APIs and projection capabilities.
2. Custom Service and CA: A self-signed CA and a service certificate/key pair were created. An echo-http-server pod was deployed, serving with this custom certificate.
3. Creating a ClusterTrustBundle Object: A ClusterTrustBundle API object was then created for the custom CA. The object's name followed a specific format derived from the signer name, and it included labels.
4. Client Pod with Projected Volume: A client pod was deployed to curl the echo-http-server. Crucially, this client pod's manifest specified a clusterTrustBundle volume, referencing the signer name and label selector for the custom CA.
5. Successful Secure Connection: The client pod successfully curled the echo-http-server using the CA certificate projected from the ClusterTrustBundle volume. Logs confirmed that the specified CA file from the volume was used for the successful TLS connection, demonstrating how easily trusted CAs can be distributed and consumed by pods without manual ConfigMap management. This also highlighted the flexibility of using label selectors to include multiple CAs for rotation.

These demos effectively illustrated how the new features address real-world security and operational challenges, providing tangible improvements to Kubernetes cluster management.

Defensive Implications

▶ Watch: Upcoming: Enhanced Image Pull Security by default (6:15)

The advancements presented by SIG Auth offer several crucial defensive implications for Kubernetes cluster operators and security teams. Adopting these features can significantly harden a cluster's security posture:

• Enforce Stronger Identity Verification: Immediately leverage Bound Service Account Token Improvements once it reaches GA. This feature's inclusion of node name and JTI in service account tokens directly counters replay attacks and strengthens the chain of identity verification, making it harder for attackers to impersonate legitimate services.
• Secure Image Supply Chain: Prioritize enabling the Image Pull Credential feature once available. This is a game-changer for preventing unauthorized access to private images cached on nodes. It mandates re-authentication for any pod attempting to use a previously pulled image without matching credentials, closing a significant security gap. Complement this by adopting Service Account Image Pull Credential to reduce reliance on long-lived imagePullSecrets by using short-lived, projected service account tokens for image pulls.
• Granular Kubelet Access: Implement Fine-grained Kubelet API Authorization to enforce the principle of least privilege. Review and update existing Role-Based Access Control (RBAC) policies for monitoring, logging, and other agents to grant access only to the specific Kubelet API endpoints (/configz, /pprof, etc.) they require, rather than broad proxy permissions.
• Streamline Certificate Management: Utilize Cluster Trust Bundles for managing trusted CAs. This simplifies certificate rotation and distribution, reducing operational overhead and the risk of misconfigurations associated with manual ConfigMap management. Consider using this for both internal service trusts and integrating with external trust anchors.
• Secure Administrative Resource Allocation: For clusters using Dynamic Resource Allocation (DRA), implement DRA Admin Access by appropriately labeling namespaces with resource.k8s.io/admin-access: "true". This ensures that critical administrative tasks requiring exclusive access to specialized hardware can proceed without impacting or being blocked by regular user workloads, preventing service disruptions.
• Enhance Key Management: Explore External Signing for Service Account Tokens to offload the management of sensitive signing keys to dedicated KMS (Key Management System) solutions. This improves key rotation, auditing, and overall key security by centralizing and securing cryptographic operations.
• Anticipate and Mitigate SSRF: Stay informed about the development of PSA Restrictions for Probes Host Fields. Once available, deploy this Pod Security Admission (PSA) control to prevent potential SSRF (Server-Side Request Forgery) vulnerabilities arising from improperly configured probe handler and lifecycle handler settings.
• Harden Node-Level Security: Monitor the progress of Harden Kubelet Server Validation and Extension of Node Restriction to Service Accounts. These features will provide additional layers of defense against Kubelet impersonation and limit the blast radius of compromised service accounts interacting with node objects.
• Consistent Identity: Leverage UID Enhancements in authenticators for better traceability and integration with external identity systems. Ensure that UIDs are consistently propagated and utilized across your security monitoring and auditing tools.

By strategically adopting these new and upcoming features, security teams can significantly reduce the attack surface, enforce stricter access controls, and build a more resilient and auditable Kubernetes infrastructure.

Key Takeaways

• Enhanced Service Account Security: Bound Service Account Token Improvements in Kubernetes v1.33 GA significantly bolster service account identity verification by embedding node name and JTI claims, effectively preventing replay attacks and strengthening the chain of trust.
• Critical Image Pulling Security Fixes: The Image Pull Credential feature addresses a long-standing vulnerability by enforcing re-authentication for images on a node if the requesting pod lacks the original pull credentials, drastically improving the default security posture of container image access.
• Simplified Certificate Management: Cluster Trust Bundles streamline the management and rotation of trusted Certificate Authorities (CAs) within the cluster by introducing a dedicated API object for signers, reducing configuration complexity and human error.
• Granular Access Control Everywhere: New features like Fine-grained Kubelet API Authorization and DRA Admin Access reinforce the principle of least privilege, allowing administrators to grant precise permissions for specific Kubelet endpoints or administrative resource claims, preventing overly broad access.
• Reduced Reliance on Long-Lived Secrets: Initiatives such as Service Account Image Pull Credential and External Signing for Service Account Tokens move towards using short-lived tokens and offloading key management to external KMS systems, mitigating risks associated with static, long-lived credentials.
• Continuous Security Evolution: SIG Auth is actively developing features like P-certificates for Service Accounts, Harden Kubelet Server Validation, and PSA Restrictions for Probes Host Fields to address emerging threats and further harden Kubernetes against SSRF and other vulnerabilities, ensuring the platform's ongoing security.

About the Speaker(s)

Rita Zhang is an engineer at Microsoft and serves as a SIG Auth co-chair within the Kubernetes community. Her work focuses on advancing authentication and authorization mechanisms in Kubernetes, contributing to the platform's security and identity management capabilities.

Stanislav Láznička is also an engineer at Microsoft and a dedicated contributor to SIG Auth. He plays a crucial role in implementing and designing many of the authentication and authorization features discussed, with a particular emphasis on practical contributions to the group's efforts. His work includes significant involvement in features like Cluster Trust Bundles and the Image Pull Credential improvement.

The speakers also acknowledged the extensive contributions of numerous community members, including Mo, James, Jordan, Vineiac, Anish, Jordan Samuel Harshel, Tahir, and Sora, highlighting that these advancements are the result of a collaborative effort across the Kubernetes community.