Containerd: Project Update and Deep Dive - M. Pavlenko, A. Suda, L. Brehm, S. Karp, J. Zhou

Containerd: Project Update and Deep Dive

Speakers: M. Pavlenko, A. Suda, L. Brehm, S. Karp, J. Zhou

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=5ZWbS01wCMk

Overview

This talk provides a comprehensive update on Containerd, the industry-standard container runtime, delivered by a panel of its core maintainers. The discussion spans recent advancements in Containerd 2.0, a preview of features slated for the upcoming 2.1 release, significant project governance changes, and a deep dive into ecosystem innovations, particularly the RunWasi project for WebAssembly workloads. The speakers, including Maxim Pavlenko, Aishiro Suda, Laura Brehm, and Shirley Karp, share their insights into the project's evolution and future direction.

The session emphasizes Containerd's critical role as a foundational component in the cloud-native landscape, powering widely adopted platforms like Docker and Kubernetes. The updates are crucial for anyone involved in container orchestration, application deployment, and infrastructure management, offering insights into improved performance, enhanced security features, and expanded extensibility. The talk highlights the ongoing efforts to stabilize APIs, address technical debt, and foster a more robust and responsive development ecosystem around Containerd.

The importance of this talk lies in its direct relevance to the stability, security, and efficiency of containerized environments. By detailing API stabilizations, new performance optimizations, and innovative runtime shims like RunWasi, the maintainers provide a roadmap for future container technology. Furthermore, the discussion on project governance, including new release cadences and a refined KEP (Kubernetes Enhancement Proposal) process, signals a commitment to improved collaboration and predictability for the broader cloud-native community.

Background

▶ Watch: Introduction to Containerd and talk agenda (0:00)

Containerd stands as a cornerstone of the cloud-native ecosystem, serving as the core container runtime for virtually all modern container platforms. Its widespread adoption is evident in its integration into Docker and its critical role within Kubernetes, including enterprise distributions and managed services. This ubiquity underscores the project's profound impact on how applications are packaged, distributed, and executed across diverse computing environments.

The project's evolution has been marked by a continuous effort to balance innovation with stability and backward compatibility. The lead-up to Containerd 2.0, which took approximately 18 months to craft since the 1.0 release, highlighted the challenges of managing a rapidly evolving codebase while serving a vast user base. A primary driver for the 2.0 release was the need to stabilize a multitude of experimental APIs introduced in previous versions, particularly 1.7. These experimental features, while forward-looking, required a period of community feedback and iterative refinement to achieve production readiness and ensure long-term viability.

Beyond API stabilization, Containerd 2.0 also aimed to address significant technical debt accumulated over time. As a project that has carried many deprecated features for backward compatibility reasons, a major release offered the opportune moment for cleanup. This involved removing legacy components and refactoring the project structure to build a more robust and maintainable foundation for future development. The process, while introducing some breaking changes, was carefully managed to minimize the blast radius for end-users, ensuring a smoother transition for the majority of the ecosystem. The ongoing need for such updates arises from the dynamic nature of container technology, where new kernel features, security paradigms, and workload types (like WebAssembly) constantly emerge, necessitating corresponding adaptations and enhancements within the runtime layer.

Key Findings

▶ Watch: Key new features introduced in Containerd 2.0 (4:00)

The talk unveiled a wealth of information regarding Containerd's current state and future trajectory, centered around several key findings and contributions:

• Containerd 2.0: Stabilization and Refinement: The latest major release, 2.0, marked a significant milestone. Its primary goals were to stabilize numerous experimental APIs introduced in 1.7, such as the Transfer Service, Sandbox API, and NRI (Node Resource Interface), making them production-ready. Concurrently, it addressed substantial technical debt by deprecating and removing legacy features like Docker schema v1 support, v1 runtimes, and the UFS snapshotter, thereby improving the project's maintainability and future extensibility. New features like Image Verifier Plugins for policy enforcement and IGZ support for faster image decompression were also integrated.
• Containerd 2.1: Upcoming Innovations: The next minor release, 2.1, promises further advancements. Notable additions include support for FS enhanced reloading file system, optimized for images with many layers and enabling image volumes for distributing AI models. It also introduces support for writing to /sys/fs/cgroup for enhanced cgroup v2 control over container resources, and more flexible user namespace UID mapping ranges.
• NerdCTL 2.1 Enhancements: The command-line client NerdCTL will see improvements in 2.1, specifically with User NS wrap mode, offering a faster alternative to rootless mode by running containers as non-root while containerd itself runs as root. An experimental Go-jail feature is also being introduced, imposing syscall restrictions on specific Go modules to mitigate supply chain attacks, drawing lessons from incidents like the XZ compression library vulnerability.
• Project Governance and Ecosystem Alignment: To improve predictability and collaboration, the project is adopting a new release cadence of minor releases every six months, starting with 2.1 in May and 2.2 in November. The LTS (Long Term Support) model has been updated, providing two years of support for LTS releases, with named volunteer maintainers. Furthermore, the KEP (Kubernetes Enhancement Proposal) process has been refined with new tracking issues and roles like "KEP shepherd" and "Sig Node liaison" to enhance visibility and communication with the Kubernetes community.
• RunWasi: WebAssembly in Kubernetes: A significant innovation in the shim layer is RunWasi, a Rust implementation of the Containerd shim designed for efficient execution of WebAssembly (Wasm) workloads in Kubernetes. It offers ready-to-use shims (e.g., Wasmtime shim, Wasmedge shim), supports OCI artifacts for Wasm modules, and includes performance optimizations like pre-compilation caching. Benchmarking demonstrates RunWasi's superior performance, being up to four times faster than runC for concurrent Wasm tasks, and enabling side-by-side execution of Linux and Wasm workloads in the same pod via Youki.

These findings collectively illustrate Containerd's ongoing commitment to performance, security, extensibility, and community collaboration, solidifying its position as a critical component of modern cloud-native infrastructure.

Technical Deep Dive

▶ Watch: Exciting upcoming features in Containerd 2.1 (6:10)

Containerd's continuous evolution is marked by sophisticated technical advancements, particularly in its API stabilization, upcoming features, and innovative shim implementations.

The Containerd 2.0 release focused heavily on solidifying experimental functionalities, transforming them into stable, production-ready components. The Transfer Service, initially introduced in 1.7, is now stable. This API leverages bidirectional streaming and data channels to robustly transfer artifacts from source to destination. While currently used for client-side image pulls, plans are underway to integrate it into the CRI (Container Runtime Interface) for Kubernetes in 2.1, promising a more efficient and resilient image pulling mechanism. The Sandbox API also achieved stability, providing a crucial abstraction layer for C-Port sandboxes. This API enables more granular control over how groups of containers are defined and managed, offering a much cleaner way to support VM-style containers. The default implementation in 2.0 utilizes pod containers as an underlying mechanism, serving as a practical example for custom implementations. Furthermore, the NRI (Node Resource Interface), a mutating webhook for container configuration, became enabled by default. NRI plugins can intercept and modify the initial container configuration generated by Containerd based on a Kubernetes PodSpec, allowing for sophisticated resource management and policy enforcement. Security was also bolstered with the introduction of Image Verifier Plugins, exec-based plugins invoked during image pulls to enforce policies, such as ensuring images originate only from trusted sources. This feature is currently integrated with the Transfer Service for client-side use, with CRI support for legacy pulls planned for 2.1. Finally, 2.0 introduced IGZ support, which significantly improves image decompression performance when available on the system, leading to faster container startup times.

Looking ahead, Containerd 2.1 brings several impactful features. A key enhancement is support for the FS enhanced reloading file system. Requiring a recent Linux kernel (6.12 or newer), this file system is optimized for images with numerous layers, where traditional overlay filesystems can become slow. Crucially, it supports image volumes, allowing an OCI image to be mounted as a human-readable volume. This is particularly useful for distributing large AI models as OCI images, enabling the separation of code and data images, allowing for easy swapping of AI models without redeploying the entire application. The 2.1 release also enhances support for cgroup v2, enabling containers to directly control their own resources such as CPU and memory by writing to /sys/fs/cgroup. Additionally, user namespace support is further enhanced, allowing for non-contiguous UID mapping ranges, providing greater flexibility in isolating container processes.

The NerdCTL command-line client also receives notable updates in version 2.1. It will support User NS wrap mode, which is a variant of rootless mode. In this configuration, containers execute as a non-root user, but Containerd itself continues to run as root. While not as strictly secure as full rootless mode, it offers a performance advantage. A significant security innovation is the experimental support for Go-jail, a mechanism to impose syscall restrictions on specific Go modules. This feature aims to mitigate potential vulnerabilities and supply chain attacks, drawing parallels to incidents like the XZ compression library attack. By restricting Go modules from executing shell commands, reading/writing arbitrary files, or creating network sockets, Go-jail reduces the attack surface, though it's not applicable to modules that rely on unsafe pointers or reflections.

Perhaps one of the most exciting technical developments discussed is RunWasi, a Rust implementation of the Containerd shim specifically designed to facilitate running WebAssembly (Wasm) workloads within Kubernetes. Wasm, a binary instruction format, promises "build once, run anywhere" capabilities, and RunWasi integrates this directly into the cloud-native stack. It provides ready-to-use shims for various Wasm runtimes, such as Wasmtime shim and Wasmedge shim. Deploying Wasm applications with RunWasi is streamlined through Kubernetes Runtime Classes, where a handler configured in the Containerd config points to the Wasmtime shim. A critical optimization in RunWasi is pre-compilation caching of Wasm modules within Containerd. This means that if the same Wasm task with the same image is run repeatedly, the Wasm module is compiled to native code only once, significantly boosting performance. Benchmarking revealed that RunWasi's Wasmtime shim can be four times faster than a distroless container running in runC for concurrent tasks, largely due to this pre-compilation technique and the fact that RunWasi embeds the Wasmtime engine, reducing memory reloading overhead. Furthermore, by leveraging Youki, another CNCF project written in Rust, RunWasi allows for the execution of both Linux containers and Wasm workloads side-by-side within the same pod, enabling patterns like service mesh sidecars for Wasm applications. The project has undergone extensive benchmarking and supports OCI artifacts for packaging and distributing Wasm modules, bringing consistency to the ecosystem.

The overall theme of extensibility was highlighted as a core strength of Containerd. Beyond shims like RunWasi, Containerd offers snapshotter extension points for managing image layers. Examples include lazy loading snapshotters like Stargz snapshotter, OverlayBD, and Nydus, which enable faster container startup by only pulling necessary parts of an image on demand. This extensibility is also leveraged in production environments, such as GKE's image streaming feature. The NRI further exemplifies this, acting as a powerful mutating webhook for dynamically adjusting container configurations, opening doors for advanced resource management and policy enforcement.

Demo / Proof of Concept

▶ Watch: Experimental Go jail for supply chain security (8:30)

The talk did not feature a live demonstration or a dedicated proof-of-concept section. The speakers primarily presented slides detailing the features of Containerd 2.0, upcoming 2.1 enhancements, project changes, and the functionalities of RunWasi. While the discussion on RunWasi included a simple YAML snippet illustrating how to configure a Kubernetes Pod with a specific runtime class to leverage the Wasmtime shim, this was for illustrative purposes rather than a live demonstration. The performance comparisons for RunWasi against runC were presented as statistical results from extensive benchmarking, rather than a real-time showcase.

Defensive Implications

▶ Watch: New release cadence and LTS model changes (10:00)

The advancements in Containerd, particularly with versions 2.0 and 2.1, and the innovations in its ecosystem like RunWasi, carry significant defensive implications for organizations operating containerized workloads. These developments offer new tools and paradigms for enhancing security, isolation, and operational resilience.

Firstly, the stabilization of the Sandbox API and its default enablement with CRI in Containerd 2.0 provides a more robust foundation for container isolation. By offering an abstraction layer for C-Port sandboxes and better support for VM-style containers, defenders can leverage this API to implement more stringent isolation boundaries, potentially mitigating the impact of container escapes or lateral movement within a compromised host. This is particularly relevant as projects like Kata Containers are actively integrating with this API to deliver microVM-based isolation.

The introduction of Image Verifier Plugins in 2.0 is a critical step towards enhancing software supply chain security. These exec-based plugins, invoked during image pulls, allow organizations to enforce policies such as pulling images only from trusted sources or requiring specific digital signatures. This capability empowers defenders to establish strong controls over the provenance and integrity of container images, directly combating risks associated with malicious or tampered images entering their environment. Integrating this with CRI in 2.1 will make it even more pervasive.

Furthermore, the experimental Go-jail feature in NerdCTL 2.1 represents a proactive defense against supply chain attacks targeting Go modules. By imposing syscall restrictions on specific Go modules, it limits their ability to perform high-risk operations like executing shell commands, reading/writing arbitrary files, or creating network sockets. This approach, inspired by incidents like the XZ vulnerability, aims to reduce the attack surface of Go-based components within the container ecosystem, even if a module is compromised. While not a silver bullet (due to limitations with unsafe pointers/reflections), it adds a valuable layer of defense.

Enhanced user namespace support in 2.1, with its ability to handle non-contiguous UID mapping ranges, improves the flexibility and effectiveness of user-based isolation within containers. This allows for more granular control over how container processes are mapped to host users, complicating privilege escalation attempts and improving the principle of least privilege.

From a performance standpoint, features like IGZ support for faster image decompression and RunWasi's pre-compilation caching for WebAssembly workloads indirectly contribute to security. Faster startup and execution reduce the window of vulnerability during container initialization and can lead to more efficient resource utilization, freeing up host resources that might otherwise be strained by inefficient processes, potentially impacting the performance of security monitoring tools.

Finally, the project's new release cadence and improved LTS model are crucial for defenders. A predictable release schedule means organizations can better plan for updates, ensuring timely application of security patches and access to new defensive features. The two-year LTS support, coupled with named volunteer maintainers, provides long-term stability and assurance for critical production deployments, reducing the risk of running unsupported software with unpatched vulnerabilities. Defenders should actively engage with these updates, migrating from deprecated features (like Docker schema v1 or legacy v1 runtimes) and adopting the new, more secure APIs and practices to maintain a robust and resilient container infrastructure.

Key Takeaways

• Foundation for Future Stability: Containerd 2.0 successfully stabilized a range of experimental APIs (Transfer Service, Sandbox API, NRI) and aggressively addressed technical debt, removing legacy features to provide a cleaner, more robust foundation for future container runtime development.
• Performance and Isolation Enhancements: Upcoming Containerd 2.1 introduces significant performance boosts (FS enhanced reloading file system for layered images, IGZ for decompression) and enhanced isolation capabilities (improved user namespaces, cgroup v2 control), crucial for modern, resource-intensive workloads.
• WebAssembly as a First-Class Citizen: RunWasi is a game-changer for WebAssembly (Wasm) in Kubernetes, offering a highly performant, Rust-based Containerd shim that leverages OCI artifacts, pre-compilation caching, and integration with Youki to run Wasm workloads efficiently and alongside traditional Linux containers.
• Strengthened Supply Chain Security: New features like Image Verifier Plugins for policy enforcement during image pulls and the experimental Go-jail in NerdCTL 2.1 demonstrate a proactive commitment to mitigating supply chain attacks and ensuring the integrity of container images and their dependencies.
• Improved Project Governance and Community Alignment: Containerd's adoption of a predictable 6-month release cadence, a clearer LTS model, and a refined KEP process with dedicated roles (KEP shepherd, Sig Node liaison) aims to foster better collaboration and predictability for the broader cloud-native ecosystem.
• Commitment to Extensibility: The talk reaffirmed Containerd's core design philosophy of being highly extensible through various extension points like shims (RunWasi, Windows, FreeBSD), snapshotters (lazy loading, image streaming), and NRI plugins, enabling a diverse range of workloads and integrations.

About the Speaker(s)

The talk featured a panel of Containerd maintainers, each contributing their unique expertise and perspective on the project's evolution:

• Joe (Moderator): Joe's involvement with container technology dates back to Docker before Containerd was split into its own project. He formally joined the Containerd project in 2017 as a security advisor, a role focused on security triage and response, and later grew into a maintainer. He has contributed to code cleanliness and testing and is particularly excited about Containerd's extensibility, highlighting its shim, snapshotter, and NRI extension points. Joe also works on GKE, where a lazy loading snapshotter powers their image streaming feature.
• Laura Brehm: Laura's journey into Containerd began by working her way down the stack, from contributing to Docker Compose to delving into engine work. Her contributions to Containerd stemmed from debugging issues in the Docker engine, leading her to become a maintainer. She is focused on ensuring shim correctness and performance, even exploring formal verification techniques to prevent regressions.
• Aishiro Suda: Aishiro is a maintainer of Youki, a Rust-based container runtime, which naturally led to his involvement with Containerd due to Youki's dependency on it. He is credited with the introduction of NerdCTL and is enthusiastic about the potential of WebAssembly for its portability, envisioning its use not only in cloud but also in desktop and mobile computing.
• Maxim Pavlenko: Maxim's first contribution to Containerd was in 2018, implementing pixie decompression to improve image decompression speeds. He is a strong advocate for Rust and its extensions within Containerd, expressing excitement about projects like RunWasi achieving production readiness with large workloads.
• Shirley Karp: Shirley has been deeply involved in integrating WebAssembly into Kubernetes. Recognizing the limitations of bundling Wasm with a full OS, she championed a lower-level integration. She was instrumental in the development of RunWasi, a project started by Brian Goff, to facilitate Wasm workloads in Kubernetes. Shirley is excited about the growing popularity of Wasm, the standardization of OCI artifact layouts for Wasm, and innovations in virtualization like Hyperlite.