A Practical Guide To Kubernetes Policy as Code - Jim Bugwadia, Rita Zhang, Andy Suderman & Joe Betz

A Practical Guide To Kubernetes Policy as Code

Speakers: Jim Bugwadia, Founder and CEO of Nurmada, Co-chair of Policy Working Group, Maintainer of Kyverno; Rita Zhang, Principal Engineer at Microsoft, Chair of SIG Auth; Andy Suderman, CTO of Fairwinds, Co-chair of Policy Working Group; Joe Betz, Staff Engineer at Google, SIG API Machinery Lead

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=w1wh9dc6m34

Overview

This talk, "A Practical Guide To Kubernetes Policy as Code," presented by a panel of distinguished experts from Google, Microsoft, Fairwinds, and Nurmada, delves into the critical role of policy in securing and managing Kubernetes environments. The speakers, all deeply involved in Kubernetes' policy and API machinery special interest groups, outline the fundamental importance of policy as code—the practice of defining and enforcing rules and conditions through codified artifacts. They emphasize that policy underpins nearly every aspect of Kubernetes operations, from security and compliance to automation and resource management.

The session meticulously explores the evolution of policy enforcement within Kubernetes, highlighting both the foundational, built-in capabilities and the advanced features offered by popular external policy engines. It provides a comprehensive comparison of Kubernetes' native Validating Admission Policy (VAP) and Mutating Admission Policy (MAP), which leverage the Common Expression Language (CEL), against established solutions like OPA Gatekeeper and Kyverno. The speakers offer practical guidance on when to utilize each approach, advocating for a hybrid strategy that maximizes efficiency and reliability while addressing complex enterprise requirements. This article aims to distill their insights into a detailed technical guide for platform engineers, security professionals, and developers navigating the intricate landscape of Kubernetes policy management.

Background

▶ Watch: Introduction to Kubernetes policy as code (0:00)

Policy has been an intrinsic part of Kubernetes since its early days, manifesting in various forms across the codebase. Examples include RBAC (Role-Based Access Control) for authorization, network policies for controlling pod-to-pod communication, resource quotas and limit ranges for resource governance, and kubelet configurations. However, the enforcement mechanisms for these policies have evolved significantly.

A key turning point came with Kubernetes 1.8, which introduced admission webhooks. These webhooks provided a foundational extension point, allowing users to intercept all write requests to the control plane and dynamically validate or mutate them before they were persisted. While powerful, admission webhooks presented several challenges. As Joe Betz pointed out, they require building a separate binary that becomes a critical component of the control plane's request serving flow. This implies significant operational overhead for extension authors, including concerns about development, maintenance, upgrade plans, scalability, and monitoring. In practice, the ecosystem observed numerous problems with admission webhooks, often leading to cluster instability if not managed meticulously.

Further analysis revealed that a significant majority of webhooks (an "80/20 split" as described by Joe Betz) were performing relatively simple validation tasks. Many developers found themselves building complex custom controllers and webhooks primarily to enforce basic validation rules at the API server's entry point. This observation led to the question: if the logic is simple, why not embed it directly into the Kubernetes API server? This line of reasoning paved the way for the adoption of the Common Expression Language (CEL). CEL was chosen for its suitability for embedding into YAML, its unsurprising C-style syntax familiar to most modern programmers, and its low execution overhead (approximately 5 to 10 times that of native code) while maintaining runtime safety and bounded execution time/memory utilization. CEL was first integrated into CRD validation rules, allowing custom resource definitions to enforce complex cross-field and immutability checks directly within their schemas, a feature successfully adopted by projects like API Gateway to eliminate their webhooks.

Key Findings

▶ Watch: Challenges with traditional Kubernetes admission webhooks (2:40)

The central theme of the talk revolves around the maturing landscape of Kubernetes policy as code, emphasizing a strategic blend of native and external enforcement mechanisms. The key findings can be summarized as follows:

1. Kubernetes Native Policy is Maturing with CEL: The introduction of Validating Admission Policy (VAP) and Mutating Admission Policy (MAP) represents a significant leap in Kubernetes' native capabilities for policy enforcement. By embedding CEL directly into the API server, these features offer a faster, more reliable, and self-contained approach to policy validation and mutation, reducing the operational overhead associated with external webhooks for common use cases.

1. CEL is the Lingua Franca for Kubernetes Policy: Across all presented solutions—native VAP/MAP, OPA Gatekeeper, and Kyverno—CEL is becoming the default or a strongly supported language for defining policy expressions. This ubiquity underscores the importance for practitioners to learn and master CEL for effective policy management in Kubernetes.

1. External Policy Engines are Adapting, Not Being Replaced: Projects like OPA Gatekeeper and Kyverno are not made redundant by native VAP/MAP. Instead, they are evolving to integrate with and leverage these native capabilities while continuing to provide advanced features that go beyond what the core Kubernetes API server can offer. This includes functionalities like background scanning, rich reporting, external data source integration, shift-left capabilities, and more complex automation.

1. Policy as Code Extends Beyond Security Validation: The speakers highlighted that policy isn't just about enforcing security constraints. It's also a powerful tool for automation, configuration generation, cleanup, and even image validation (e.g., checking signatures and attestations). This broader scope empowers platform engineers to build self-service platforms and standardize configurations.

1. Hybrid Approach is Optimal: The consensus among the speakers is that there isn't a single "best" policy solution. A practical strategy involves using native VAP/MAP for straightforward, performance-critical validations and mutations, and then extending these capabilities with external webhooks for more complex, advanced, or ecosystem-specific requirements. This allows organizations to optimize for both reliability/performance and feature richness.

1. Mutation Requires Caution: While powerful for automation, mutating admission policies (whether native or via webhooks) carry inherent risks and can be expensive. The advice is to use mutation judiciously and minimize its application, focusing on scenarios where it provides significant value and is carefully managed.

Technical Deep Dive

▶ Watch: Introduction of Common Expression Language (CEL) (4:30)

The talk provides a granular look into the technical mechanisms behind Kubernetes policy as code, showcasing the evolution from custom webhooks to built-in CEL-powered policies and the complementary roles of external policy engines.

Kubernetes Native Policy with CEL (Joe Betz)

Joe Betz elaborated on the direct integration of Common Expression Language (CEL) into the Kubernetes API server, marking a significant step towards native policy enforcement.

1. CRD Validation Rules: The initial and highly successful application of CEL was in Custom Resource Definition (CRD) validation rules. These rules allow CRD authors to embed CEL expressions directly into their schema, enabling complex cross-field validations (e.g., ensuring minReplicas is always less than maxReplicas) and immutability checks (e.g., preventing changes to a field after creation). This feature has allowed large projects, such as API Gateway, to migrate away from custom webhooks, simplifying their architecture and improving reliability.

1. Validating Admission Policy (VAP): VAP is a direct, in-tree substitute for validating admission webhooks. It allows cluster administrators to define policies using CEL expressions without deploying any external binaries. A VAP configuration involves three key resources:

• ValidatingAdmissionPolicy: Defines the policy logic using a CEL expression within spec.validations.expression. This expression evaluates to true for valid requests and false for invalid ones.
• ValidatingAdmissionPolicyBinding: Binds the policy to specific resources (e.g., Pods, Deployments) and namespaces within the cluster. This allows for granular control over where a policy is enforced.
• ParamKind (Optional): An optional resource that allows policies to be configured with parameters, enabling reuse of policy logic with different enforcement values (e.g., setting a maxReplicaLimit to 3 for a specific binding).

VAP also supports advanced authorization checks within CEL expressions. For instance, a policy can check if a user attempting to change a specific label on a resource has permission to perform a custom verb (e.g., set-privileged-label), leveraging the existing RBAC system. This enables fine-grained control over privileged operations.

1. Mutating Admission Policy (MAP): (Currently in Alpha) MAP extends the concept of VAP to allow for object mutation. Instead of simply validating a request, MAP policies return a JSON patch that modifies the incoming object. An example provided was the injection of a sidecar container into a Pod definition. While powerful for automation, the alpha status and inherent complexities of mutation (discussed later by Andy Suderman) mean it requires careful consideration.

OPA Gatekeeper (Rita Zhang)

Rita Zhang introduced OPA Gatekeeper, a widely adopted open-source project from the CNCF, built on the Open Policy Agent (OPA) rule engine.

1. OPA Foundation: OPA, a CNCF graduated project (since 2021) and celebrating its 10-year anniversary, serves as a general-purpose policy engine. Gatekeeper brings OPA's capabilities into the Kubernetes ecosystem as a dynamic, flexible admission and mutation webhook, CLI, and controller. The core principle is "write policy once as code and configurations, and run it anywhere."

1. Key Differentiators:

• Multiple Languages and Engines: Gatekeeper started with Rego, OPA's native policy language, but has recently added full CEL support. This allows users to choose their preferred language and leverage different engines, including OPA itself and Kubernetes VAP.
• Multi-Target Enforcement: The same policy code can be applied across various targets beyond Kubernetes, such as Terraform, demonstrating its versatility for broader infrastructure as code scenarios.
• Separation of Concerns: Gatekeeper decouples policy logic from its configuration. ConstraintTemplates define the policy rules (e.g., "no root containers"), while Constraints apply these rules with specific parameters (e.g., "enforce no root containers in namespace dev"). This separation allows different personas (policy authors vs. deployers) to manage their respective resources effectively.
• Extensive Policy Library: Gatekeeper boasts a rich community-contributed policy library, offering ready-to-use policies. Notably, this library was instrumental in assisting users migrate from deprecated Pod Security Policies (PSPs).

1. Advanced Scenarios Beyond VAP/MAP: Rita highlighted scenarios where Gatekeeper excels beyond native Kubernetes policies:

• Audit: Gatekeeper can audit existing cluster resources for policy violations, streaming data for compliance or tracing.
• Shift-Left: The gator CLI tool enables policy testing in CI/CD pipelines, allowing developers to catch violations before deployment.
• Metrics: Integrates with Prometheus for monitoring policy enforcement and violations.
• Context Awareness/Referential Checks: Policies can inspect the state of other resources already in the cluster (e.g., ensuring uniqueness of Ingress hostnames).
• External Data Sources: Policies can interact with external services or data sources during admission.
• Pub/Sub for Violations: Allows subscribing to and streaming violation events.

1. Gatekeeper and VAP Integration: Gatekeeper aims to be a "front-end to unify the experience" for users. It can generate native VAP resources for common CEL-based policies, allowing the in-tree Kubernetes admission controller to handle enforcement. For advanced scenarios (e.g., audit, referential checks, external providers), Gatekeeper's own webhook takes over. This hybrid approach allows users to write policies once (in CEL or Rego) and have Gatekeeper intelligently delegate to the most appropriate enforcement mechanism.

1. Advanced Mutation with Gatekeeper: While recommending native MAP for basic mutations, Gatekeeper offers advanced mutation capabilities through its webhook, particularly for scenarios involving external data providers. It's designed for global idempotency, ensuring predictable results even with multiple or re-invoked mutations, and can make parallel calls to external services, crucial for minimizing latency during admission.

Kyverno (Jim Bugwadia)

Jim Bugwadia presented Kyverno, an alternative policy engine that started five years ago with a distinct philosophy: policy as code for a full lifecycle of automation, not just validation.

1. Comprehensive Policy Toolbox: Kyverno's mission is to simplify policy management for Kubernetes by providing a toolbox that includes mutation, generation, cleanup, and image validation, alongside validation. This holistic approach aims to reduce the need for additional custom controllers, offering a standardized solution for platform engineers building self-service platforms.

1. Kubernetes-Native Design: Kyverno's initial focus was to make policy management simple for Kubernetes administrators, minimizing the need to learn new languages or complex extensions. While initially supporting JamesPath, Kyverno now fully supports CEL.

1. Evolution and Delegation: Similar to Gatekeeper, Kyverno is evolving to delegate to native Kubernetes policies (VAP, MAP) wherever possible. Kyverno policies written with CEL expressions can automatically generate VAP and MAP resources for in-line execution, leveraging the API server's efficiency. Kyverno's own webhooks continue to handle its unique extension features.

1. Kyverno 1.14 Features: The latest release, Kyverno 1.14, introduces five new policy types, furthering its evolution. Jim specifically highlighted:

• Validating Policy (Kyverno's Extension of VAP): This is Kyverno's flavor of VAP, extending the native ValidatingAdmissionPolicy with additional tags for features like evaluationMode (for Kubernetes resources or generic JSON/YAML), webhookConfiguration (for timeouts), and autoGeneration (for Pod controllers, VAP, and MAP). Kyverno also provides an enhanced CEL environment that supports advanced functions, such as fetching data from OCI registries to check manifest configurations or image parsing functions, some of which are being proposed upstream to Kubernetes.
• Image Validating Policy: This powerful feature allows Kyverno to work with Notary and Cosign to check for image signatures and attestations. Policies can verify if an OCI image has been scanned for vulnerabilities, if an SBOM (Software Bill of Materials) exists, what vulnerabilities are present, and if the image was signed by authorized entities. This policy type is highly flexible, applicable to Kubernetes resources, or any JSON/YAML payload, including serverless container images, with minimal changes.

1. Kyverno's Unique Advantages over VAP: Jim outlined key areas where Kyverno provides additional value:

• Background Scans: Policies can be applied not just at creation time but also retroactively to existing resources, crucial for auditing and compliance.
• Fine-Grained Exceptions: Kyverno offers granular exception management, for example, excluding specific images within a container from a policy.
• Standardized Reporting: Kyverno supports the Policy Working Group report format, providing consistent reporting across namespaces and clusters.

These features are also supported for the built-in VAP and MAP when managed by Kyverno.

Kyverno's mission is to make policy universally usable across pipelines, admission controllers, and APIs, fostering collaboration among developers, operators, and security teams using standard Kubernetes tooling.

Demo / Proof of Concept

▶ Watch: CEL for CRD validation rules in Kubernetes (6:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, the speakers effectively illustrated their points with concrete code examples embedded within their slides. These examples served as direct proofs of concept for the discussed policy types and capabilities.

Joe Betz showcased:

• A CEL expression for CRD validation rules, demonstrating cross-field validation (e.g., minReplicas <= maxReplicas).
• A complete Validating Admission Policy configuration, including the ValidatingAdmissionPolicy resource with its CEL expression, the ValidatingAdmissionPolicyBinding to apply it, and an optional ParamKind for parameterization.
• A CEL expression within VAP performing authorization checks, verifying if a user has a specific RBAC permission before allowing a label change.
• An example of a Mutating Admission Policy (in alpha) showing how a JSON patch could inject a sidecar container.

Rita Zhang presented:

• A Gatekeeper ConstraintTemplate that supports multiple engines, explicitly showing both a CEL expression for Kubernetes validating policy and a Rego policy for advanced scenarios. This highlighted Gatekeeper's flexibility in language choice and its ability to generate VAP resources.

Jim Bugwadia provided:

• A Kyverno Validating Policy declaration, illustrating its extensions to the native VAP with fields like evaluationMode, webhookConfiguration, and autoGeneration.
• An example of a Kyverno Image Validating Policy using CEL expressions to check OCI image signatures, attestations for vulnerabilities, and SBOM presence. He also highlighted the ease of adapting such a policy from Kubernetes resources to generic JSON payloads with a "two-line change."

These embedded code snippets and configuration examples provided clear, practical insights into how each policy solution is defined and operates, serving as strong illustrative proofs of concept for their respective capabilities.

Defensive Implications

▶ Watch: Validating Admission Policy as a declarative solution (6:40)

The detailed exploration of Kubernetes policy as code offers several critical implications for defenders striving to secure and manage their clusters effectively:

1. Prioritize Native Policies for Baseline Security: For common and straightforward validation or mutation requirements, defenders should leverage Kubernetes' native Validating Admission Policy (VAP) and Mutating Admission Policy (MAP). These built-in features, powered by CEL, offer superior performance, reliability, and reduced operational overhead compared to running external webhooks for simple tasks. They should form the first line of defense for basic security hygiene, such as enforcing specific label presence, preventing certain resource types, or ensuring immutability of critical fields.

1. Master CEL as a Core Skill: The pervasive adoption of Common Expression Language (CEL) across native Kubernetes policies, OPA Gatekeeper, and Kyverno makes it an indispensable skill. Defenders must invest in learning CEL to effectively write, understand, and troubleshoot policy definitions, whether for in-tree enforcement or as part of external policy engines.

1. Strategically Adopt External Policy Engines for Advanced Needs: While native policies are foundational, external solutions like OPA Gatekeeper and Kyverno are crucial for advanced defensive postures.

• Auditing and Compliance: Use Gatekeeper or Kyverno for background scanning of existing resources to detect drift or violations, stream audit data to SIEMs, and ensure continuous compliance.
• Shift-Left Security: Integrate tools like Gatekeeper's gator CLI into CI/CD pipelines to empower developers to detect policy violations early, reducing the cost and risk of fixing issues in production.
• Image Supply Chain Security: Leverage Kyverno's Image Validating Policy to enforce strong controls over container images. This includes verifying image signatures (Cosign, Notary), checking for attestations (e.g., vulnerability scan results, SBOM presence), and ensuring images originate from trusted registries.
• Context-Awareness and External Data: For policies requiring insights into the broader cluster state (e.g., uniqueness checks) or interaction with external data sources (e.g., threat intelligence feeds, asset inventories), Gatekeeper's and Kyverno's advanced capabilities are essential.
• Comprehensive Automation: Kyverno's generation, mutation, and cleanup features can automate security best practices, such as injecting security-hardened sidecars or enforcing specific resource configurations across the cluster.

1. Exercise Caution with Mutation: Mutating admission webhooks, whether native MAP or through external engines, should be used sparingly and with extreme caution. As highlighted, they are "expensive and risky." Uncontrolled mutations can lead to unpredictable behavior, break GitOps workflows, or introduce security vulnerabilities. Defenders should favor validation over mutation where possible and, when mutation is necessary, ensure it's idempotent, well-tested, and tightly scoped using match conditions to minimize invocation overhead.

1. Integrate with GitOps Workflows: For organizations adopting GitOps, it's crucial to understand how policy enforcement interacts with tools like Argo CD and Flux. The talk mentioned that these tools now support server-side apply, which helps them coexist with mutating webhooks by allowing the API server to manage object changes, rather than the GitOps tool constantly trying to revert them. This integration is vital for maintaining a single source of truth and preventing reconciliation loops.

1. Establish Clear Roles and Responsibilities: The separation of concerns offered by Gatekeeper (ConstraintTemplate vs. Constraint) and Kyverno (policy definition vs. exclusion rules) provides a framework for defining distinct roles for policy authors (security teams, platform engineers) and policy deployers (cluster administrators). This ensures that policy logic is consistent, while its application can be tailored to specific cluster environments or namespaces.

By strategically combining Kubernetes native policies with the advanced features of external engines, mastering CEL, and adopting a cautious approach to mutation, defenders can build robust, scalable, and adaptable security and governance frameworks for their Kubernetes environments.

Key Takeaways

• Kubernetes Native Policies are Evolving Rapidly: The introduction of Validating Admission Policy (VAP) and Mutating Admission Policy (MAP) with Common Expression Language (CEL) provides fast, reliable, and self-contained policy enforcement directly within the Kubernetes API server for common scenarios.
• CEL is a Fundamental Skill: CEL is becoming the standard expression language across Kubernetes' native policy features and is widely adopted by external policy engines like OPA Gatekeeper and Kyverno. Learning CEL is crucial for anyone involved in Kubernetes policy management.
• External Policy Engines Offer Advanced Capabilities: Solutions like OPA Gatekeeper and Kyverno complement native policies by providing advanced features such as background scanning, rich reporting, integration with external data sources, shift-left capabilities (e.g., gator CLI), comprehensive image validation (signatures, attestations), and complex automation (generation, cleanup).
• Adopt a Hybrid Policy Strategy: The most effective approach involves leveraging native VAP/MAP for basic, performance-critical validation and mutation, while strategically using external webhooks for more complex, context-aware, or ecosystem-specific requirements.
• Exercise Caution with Mutation: Mutating admission policies, whether native or via webhooks, are powerful but can be expensive and risky. Their use should be minimized, carefully designed for idempotency, and tightly scoped to specific, high-value automation scenarios.
• Policy as Code Extends Beyond Security: Policy encompasses not just security validation but also automation, configuration generation, resource cleanup, and image supply chain security, enabling platform engineers to build robust, self-service Kubernetes platforms.

About the Speaker(s)

The talk was delivered by a panel of four distinguished experts, each deeply involved in the Kubernetes community and policy-related initiatives:

• Andy Suderman is the CTO of Fairwinds and serves as a co-chair of the Kubernetes Policy Working Group. His expertise lies in helping organizations achieve security and reliability in their Kubernetes environments.
• Joe Betz is a Staff Engineer at Google and holds the lead position for the Kubernetes SIG API Machinery. His work focuses on the core components and extensibility of the Kubernetes API server, including the integration of CEL.
• Rita Zhang is a Principal Engineer at Microsoft and the chair of the Kubernetes SIG Auth, which oversees authentication, authorization, and policy enforcement within Kubernetes. She has been instrumental in bringing enterprise rule engine capabilities to the Kubernetes ecosystem.
• Jim Bugwadia is the Founder and CEO of Nurmada, a co-chair of the Kubernetes Policy Working Group, and a maintainer of Kyverno. His work centers on simplifying policy management and extending its capabilities for Kubernetes and broader cloud-native environments.