Demystifying Why the World Is Built on Kubernetes: Learning To Lev... Abby Bangser & Sebastien Blanc

Demystifying Why the World Is Built on Kubernetes: Learning To Lev... Abby Bangser & Sebastien Blanc

Speakers: Abby Bangser, Principal Engineer, Centaso; Sebastien Blanc, Developer Advocate, Port

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=dVM20108SRc

Overview

In this insightful KubeCon EU talk, Abby Bangser and Sebastien Blanc delve into the often-overlooked developer persona of Kubernetes, arguing that its true power lies in its extensibility as an API server, rather than solely as a container orchestrator. Titled "Demystifying Why the World Is Built on Kubernetes: Learning To Lev...", the presentation aims to clarify the foundational concepts of Custom Resource Definitions (CRDs) and controllers, which are pivotal for extending Kubernetes' capabilities. The speakers emphasize that understanding these mechanisms is crucial for organizations looking to build sophisticated internal developer platforms and leverage Kubernetes beyond its out-of-the-box functionalities.

The talk addresses the common confusion surrounding Kubernetes' various interaction models and the plethora of associated acronyms. By breaking down CRDs and controllers into understandable components and debunking prevalent myths, Bangser and Blanc empower developers to harness Kubernetes for creating tailored, self-service solutions. Their core message is that Kubernetes provides a powerful, single control plane for managing diverse infrastructure, both within and outside the cluster, making it an ideal foundation for modern platform engineering initiatives. This perspective transforms Kubernetes from a mere container scheduler into a robust framework for defining and automating complex operational workflows.

Background

▶ Watch: Kubernetes interaction models and talk focus (2:15)

Kubernetes is a multifaceted system, often perceived differently by various stakeholders within an organization. Abby Bangser introduced three distinct interaction models, or "personas," to highlight this divergence:

1. The User: This persona interacts with Kubernetes primarily to manage applications. They use tools like kubectl to deploy, monitor logs, and ensure their applications (pods) are running as expected. Their focus is on the operational state of their software.
2. The Administrator: More infrastructure-oriented, administrators manage the Kubernetes cluster itself. Their responsibilities include creating nodes, handling taints, performing Kubernetes upgrades, and setting cluster-wide standards such as Role-Based Access Control (RBAC). They view Kubernetes as a container scheduler and a hosting solution for various CNCF tools.
3. The Developer: This is the least discussed, yet arguably the most impactful, persona in the context of extending Kubernetes. Developers, often with backgrounds as users or administrators, see Kubernetes as a powerful API server. Their goal is to extend Kubernetes' native capabilities to bring more value to their organizations, building custom functionalities that go beyond simple container orchestration.

This divergence in perception often leads to confusion, as different teams speak about Kubernetes from fundamentally different vantage points. The speakers clarified that all these perspectives are valid, contributing to Kubernetes' widespread adoption. However, their talk specifically focuses on the developer persona, aiming to demystify how one can leverage Kubernetes as an API server to build custom solutions. The underlying problem addressed is the inherent complexity and jargon associated with extending Kubernetes, which can be a significant barrier to entry for many developers. By simplifying the understanding of CRDs and controllers, the talk provides a foundational entry point into this powerful aspect of Kubernetes.

Key Findings

▶ Watch: Extending Kubernetes API with custom resources (4:05)

The central findings of the talk revolve around the transformative power of Custom Resource Definitions (CRDs) and controllers in extending Kubernetes.

Firstly, CRDs are presented as the fundamental building blocks for extending the Kubernetes API. They allow developers to define new, custom types of resources that Kubernetes can then manage. This capability elevates Kubernetes beyond a mere container orchestrator, enabling it to model and manage virtually any domain-specific concept within an organization's infrastructure. The speakers highlighted that this ability to define custom "kinds" is one of Kubernetes' most powerful features, allowing it to serve as a versatile API server.

Secondly, controllers are introduced as the active agents that bring CRDs to life. A controller is an event-driven application, typically running as a pod within the Kubernetes cluster, that watches for changes to specific resource types (including custom ones) and then takes action to reconcile the desired state with the actual state. This reconciliation loop is the core mechanism by which Kubernetes automates and maintains the desired configuration of resources. The talk illustrated how controllers can chain actions, where one controller's output (e.g., a new resource) can become the input for another controller.

Beyond these core definitions, the speakers dedicated a significant portion of their talk to debunking common myths surrounding Kubernetes extensibility:

1. "Controllers are only for Kubernetes resources." This is false. Controllers can manage resources outside of Kubernetes, such as cloud provider services (demonstrated by projects like Crossplane), bare metal, mainframes, or VMs. Kubernetes provides a single control plane, but its influence extends far beyond the cluster boundary.
2. "Controllers can only watch Kubernetes resources." While more advanced, controllers can also be configured to watch non-Kubernetes resources, adding another layer of integration for complex, hybrid environments.
3. "Golang is the only way to work with Kubernetes and write controllers/operators." This is a pervasive myth. While Kubernetes itself is built in Go, and many early operators were Go-based, controllers are just applications running in a pod. Developers can use virtually any language, with robust SDKs available for Rust (kube-rs), Python (Pythonic framework), Node.js (NodeJS operator framework), and Java (Java Operator SDK). Sebastien Blanc particularly praised the Java Operator SDK for its developer-friendly experience.
4. "You need to know everything to get started." The speakers emphasized that complexity should not be a barrier. They clarified the nuanced relationship between controllers (generic pod reacting to a resource) and operators (a specialized type of controller focused on lifecycle management of specific software, often following Red Hat's Operator Framework capability model). While the distinction is important, new developers should not be deterred by it and can start with small, simple controllers.
5. "Building on top of Kubernetes is only for vendors." This is unequivocally false. Organizations of all sizes can and should build their own custom operators and CRDs if it makes sense for their internal needs. This is particularly relevant for platform engineering initiatives, where CRDs become the domain model and APIs for Internal Developer Platforms (IDPs), offering self-service capabilities to developers.

These findings collectively highlight Kubernetes' architectural elegance and its potential to serve as a universal control plane, driven by custom logic and domain-specific APIs.

Technical Deep Dive

▶ Watch: OpenAPI schema for CRD field validation (6:00)

At the heart of Kubernetes' extensibility are Custom Resource Definitions (CRDs) and controllers. Understanding their technical underpinnings is crucial for leveraging Kubernetes as a powerful API server.

Custom Resource Definitions (CRDs)

A CRD is a Kubernetes API resource that allows you to define a new, custom resource kind. It essentially extends the Kubernetes API schema. When a CRD is applied to a cluster, the Kubernetes API server begins to accept and store objects of that new kind, just like it does with native resources such as Pods or Deployments.

Technically, a CRD is a YAML manifest that typically includes:

• apiVersion: Specifies the Kubernetes API version (e.g., apiextensions.k8s.io/v1).
• kind: Always CustomResourceDefinition.
• metadata: Standard Kubernetes object metadata (name, labels, etc.).
• spec: This is where the custom resource's schema is defined.

The spec section is critical, as it uses an Open API v3 schema to describe the structure and validation rules for the custom resource. This schema defines:

• Field types: Whether a field is a string, integer, boolean, array, or object.
• Constraints: Minimum/maximum values for numbers, string patterns, or length constraints.
• Required fields: Marking certain fields as mandatory.
• Enums: Limiting string fields to a predefined set of values (e.g., dev or prod for an environment field).
• Defaults: Providing default values for fields if not specified.

For example, a CRD for a PostgresDatabase might define fields like name (string, required), size (integer, with min/max constraints), and environment (string, enum of dev or prod). If a user attempts to create a custom resource (an instance of the CRD) that violates these schema rules, the Kubernetes API server will reject it, ensuring data integrity within etcd, Kubernetes' key-value store. Sebastien Blanc drew a helpful analogy: a CRD is like a Java class definition (or a POJO), while a Custom Resource (CR) is an instance or object created from that class definition. Similarly, one could think of a CRD as a database schema and a CR as a row in that table.

Controllers

While CRDs define what a custom resource looks like, controllers define how Kubernetes reacts to instances of these custom resources. A controller is essentially an event-driven application that runs as a pod within the Kubernetes cluster. Its primary function is to:

1. Watch for changes to specific resource types (native or custom).
2. Reconcile the observed actual state of the cluster with the desired state specified in the watched resource.
3. Take action to bridge any gaps, which could involve creating, updating, or deleting other resources (Kubernetes native or external).

The core of a controller's logic is its reconciliation loop. This loop continuously checks if the current state matches the desired state. If a discrepancy is found (e.g., a PostgresDatabase CR specifies a database that doesn't exist yet), the controller executes a series of actions to achieve the desired state (e.g., provision a PostgreSQL instance, create a user, set up networking). This pattern is fundamental to Kubernetes' self-healing and automation capabilities.

A common example of a native controller is the Deployment controller. When a user creates a Deployment resource, the Deployment controller watches this CR. Its to-do list includes creating a ReplicaSet resource. The ReplicaSet controller then watches the ReplicaSet, and its to-do list involves creating individual Pod resources. This demonstrates how controllers can form a chain, each responsible for a specific, well-defined task, adhering to the Unix philosophy of doing one thing well.

Custom Controllers and Abstraction

The real power emerges when developers write custom controllers for their CRDs. Abby Bangser illustrated this with Kratix, a platform that uses custom CRDs and controllers to provide platform services. In Kratix, a Promise is a custom resource that defines a platform service (e.g., a managed database). The Kratix controller watches Promise CRs. When a Promise is created, the Kratix controller's to-do list includes:

1. Creating another CRD (e.g., a Service API CRD) that allows users to request instances of the promised service.
2. Creating a backend controller specifically for that new Service API CRD. This backend controller is then responsible for provisioning and managing the actual service (e.g., calling a cloud provider API to create a database instance).

This example highlights how CRDs and controllers enable the creation of powerful abstractions, allowing organizations to define their own platform APIs and automate complex provisioning workflows, both on-cluster and by integrating with external systems. The flexibility of controllers to manage off-cluster resources (as seen in tools like Crossplane, which uses controllers to provision cloud infrastructure) is a testament to Kubernetes' role as a universal control plane.

Demo / Proof of Concept

▶ Watch: CRD as Java class, custom resource as object instance (6:35)

The talk did not feature a live, interactive demonstration in the traditional sense. However, the speakers provided strong conceptual proofs of concept through detailed examples and architectural diagrams.

Abby Bangser used the Kratix platform, which she helps build, as a prime example of how custom CRDs and controllers are leveraged to create higher-level abstractions. The description of how a Promise CRD leads to the dynamic creation of a Service API CRD and an associated backend controller effectively illustrates the power of this extensibility model. This serves as a robust conceptual demonstration of how organizations can define their own platform services and automate their provisioning and lifecycle management through Kubernetes.

Furthermore, Sebastien Blanc's enthusiastic endorsement of the Java Operator SDK provided a practical proof of concept for developers. He highlighted that this SDK significantly simplifies the process of building controllers, automating tasks like converting Java POJOs into CRDs and managing the event loop. This example serves to show that developing custom controllers is not an arcane art reserved for Go experts but is accessible to developers across various language ecosystems, fostering what he termed "developer joy." While not a live demo, these examples effectively convey the capabilities and benefits of CRDs and controllers in real-world platform engineering scenarios.

Defensive Implications

▶ Watch: CRDs define state, but need controllers for action (8:20)

While this talk was not explicitly focused on security, the principles of CRDs and controllers have significant implications for a robust defensive posture within a Kubernetes environment, especially when building Internal Developer Platforms (IDPs).

1. Enhanced Validation and Data Integrity: CRDs, with their embedded Open API v3 schema definitions, provide a powerful mechanism for input validation. By strictly defining field types, constraints (e.g., minimum/maximum values, regex patterns), and required fields, organizations can prevent the application of malformed or potentially malicious custom resources. This acts as a crucial first line of defense against configuration errors and attempts to inject unauthorized or invalid parameters into the system, ensuring the integrity of data stored in etcd.

1. Granular Access Control: Custom resources are subject to Kubernetes' Role-Based Access Control (RBAC). This means that administrators can define precise permissions for who can create, read, update, or delete specific custom resource types and instances. For defenders, this is critical for limiting the blast radius of compromised accounts or applications. By segmenting access to custom resources, organizations can ensure that only authorized teams or service accounts can manipulate the custom logic and infrastructure managed by custom controllers.

1. Supply Chain Security for Custom Logic: When custom controllers are introduced, they become part of the organization's software supply chain. Defenders must ensure that the container images used for controllers are sourced from trusted registries, scanned for vulnerabilities, and signed. If controllers interact with external APIs or pull configurations from external sources, the security of those integrations (authentication, authorization, data encryption in transit) must be rigorously vetted. This extends the security perimeter beyond native Kubernetes components to include all custom extensions.

1. Observability and Anomaly Detection: Custom controllers introduce new operational logic into the cluster. Comprehensive monitoring and logging of controller activities are essential. Defenders should track reconciliation loop failures, unexpected resource creations or deletions, and API calls made by controllers, especially those interacting with external systems. Anomalies in controller behavior could indicate misconfigurations, bugs, or even malicious activity, necessitating robust alerting and incident response capabilities tailored to custom resources.

1. Principle of Least Privilege: Custom controllers, like any other application, should operate with the minimum necessary permissions. Their Service Accounts should be scoped to only interact with the specific CRDs and native resources they are designed to manage. Over-privileged custom controllers could be exploited to gain broader access to the cluster or external infrastructure. Implementing this principle reduces the potential impact of a compromised controller.

1. Structured Platform Governance: The talk emphasizes using CRDs as the domain model for an IDP. This approach brings structure and standardization to platform capabilities. For defenders, a well-defined and versioned API for internal platform services, backed by CRDs, is easier to audit, secure, and govern than ad-hoc scripts or disparate tools. It allows security policies to be consistently applied across all platform offerings, streamlining compliance and reducing the attack surface.

By actively considering these defensive implications during the design and implementation of CRDs and controllers, organizations can build secure, resilient, and manageable Kubernetes-based platforms.

Key Takeaways

• Kubernetes as an API Server: Beyond container orchestration, Kubernetes' most powerful feature is its extensibility as a robust API server, enabling custom domain modeling and automation.
• CRDs Extend the API: Custom Resource Definitions allow developers to define new resource types with strong schema validation, extending Kubernetes' native capabilities for specific organizational needs.
• Controllers Drive Automation: Controllers are event-driven applications that watch resources (including custom ones) and continuously reconcile the desired state with reality, automating complex operational workflows.
• Unified Control Plane: The Kubernetes control plane, powered by CRDs and controllers, can manage both on-cluster resources and external infrastructure (e.g., cloud services, bare metal), providing a single operational interface.
• Language Agnostic Development: Custom controllers are not limited to Golang; robust SDKs exist for various languages like Java, Python, and Rust, making Kubernetes extensibility accessible to a broader developer base.
• Foundation for IDPs: CRDs and controllers are fundamental building blocks for creating powerful, self-service Internal Developer Platforms (IDPs), allowing organizations to abstract complex infrastructure and offer streamlined developer experiences.

About the Speaker(s)

Abby Bangser is a Principal Engineer at Centaso, where she is actively involved in building a framework for platforms, with a strong focus on making Kubernetes and its advanced functionalities more accessible to a wider audience. Her work at Centaso centers on creating tools and abstractions that simplify the complexities of Kubernetes for platform builders. Abby has hands-on experience building controllers and operators, notably contributing to Kratix, a platform that leverages these concepts to define and deliver platform services.

Sebastien Blanc, known as Sebie, is a Developer Advocate for Port, a company that provides an internal developer portal. In his role, Sebastien helps developers understand and utilize complex technologies, including Kubernetes extensibility. As a Java developer, he has a particular affinity for the Java Operator SDK, which he praises for significantly enhancing the developer experience when building custom controllers. Sebastien also has practical experience in building and advocating for controllers and operators, including Port's own connector that integrates Kubernetes clusters into their portal.