Get WITty: Evolving Kubernetes Scheduling With the WebAssembly... Dejan Pejchev & Jonathan Giannuzzi

Get WITty: Evolving Kubernetes Scheduling With the WebAssembly Component Model

Speakers: Dejan Pejchev, Open Source Engineer, G Research; Jonathan Giannuzzi, Open Source Engineer, G Research

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=hjbZOBghxYU

Overview

In this insightful KubeCon EU talk, Dejan Pejchev and Jonathan Giannuzzi of G Research explore their journey to enhance the Kubernetes scheduler by leveraging the WebAssembly Component Model. The talk addresses a significant challenge in extending Kubernetes: the difficulty of writing custom scheduling logic in multiple programming languages efficiently and maintainably. Traditional extension methods often involve high latency, language restrictions, or complex manual memory management, hindering broad adoption and development agility.

The speakers meticulously detail the limitations of current WebAssembly (WASM) plugins for Kubernetes, which are predominantly Go-centric and require intricate manual memory marshaling. They propose the WebAssembly Component Model as a revolutionary solution, promising a standardized, language-neutral interface for defining complex data types and automatically generating language bindings. This approach aims to liberate developers from the burdens of boilerplate code and language-specific glue, enabling them to write scheduler plugins in their preferred languages with native-like performance.

This presentation is crucial for anyone involved in Kubernetes infrastructure, custom resource management, or high-performance cloud-native applications. It highlights the potential of the WebAssembly Component Model to transform how extensible systems are built, offering a glimpse into a future where complex cross-language interoperability is seamless and efficient. The talk not only showcases the ambitious vision but also candidly shares the significant technical hurdles encountered, particularly the immaturity of the Component Model's tooling and Go runtime support, advocating for community collaboration to accelerate its development.

Background

▶ Watch: Introduction: Evolving Kubernetes Scheduling with WebAssembly (0:00)

The Kubernetes scheduler is a critical component responsible for assigning pods to nodes. Its extensibility is vital for tailoring cluster behavior to specific workloads, resource requirements, and organizational policies. Jonathan Giannuzzi began by briefly revisiting the Kubernetes scheduling cycle and its numerous extension points, which were the subject of their previous KubeCon talk. He outlined three primary methods for extending the scheduler:

1. Scheduler Extenders: These are essentially webhooks that communicate over HTTP with external services. While language-agnostic, their network-based communication introduces significant latency and they offer only four extension points, making them a legacy solution for high-performance or fine-grained control.
2. Scheduling Framework Plugins: This is the "big boy stuff," where custom code becomes an integral part of the scheduler binary itself. Written in Go, these plugins are highly efficient due to their native execution but require deploying a custom scheduler binary, which can be inflexible for updates and management. They offer over a dozen extension points, providing extensive control.
3. WASM Plugins: Introduced as a more flexible alternative, WebAssembly offers portability across architectures (ARM, Intel), sandboxed execution, and multi-language support. In the context of Kubernetes, a specific kube-scheduler WASM extension acts as a framework plugin, forwarding calls to user-defined WASM modules. These modules run via Wazero, a zero-dependency Go WASM runtime. However, a significant limitation was highlighted: current WASM plugins could only be written in "tiny Go," a restricted subset of the Go language, severely limiting developer choice and broader adoption.

The core problem stems from the low-level nature of WASM's host-guest interface. When the Go host (the Kubernetes scheduler extension) needs to call a WASM guest module, it can only pass primitive types like pointers and integers on the stack. Complex data structures, such as a Kubernetes Pod definition or NodeInfo, must be manually serialized, passed as byte arrays, and then deserialized within the guest module. This involves "memory shenanigans" – intricate buffer management, string copying, and pointer manipulation – which is tedious, error-prone, and necessitates language-specific SDKs for each guest language, making maintenance a nightmare. The existing Go SDK for WASM plugins exemplifies this complexity, requiring developers to deal with mem.get_string and similar low-level memory operations, abstracting away the underlying complexity only partially.

This manual, language-specific glue code for complex types is precisely the problem the WebAssembly Component Model aims to solve. The Component Model promises a canonical API, a standardized low-level binary interface that dictates how complex data types are represented, serialized, and how references are managed across language boundaries. This would theoretically allow for the automatic generation of language bindings (SDKs) for both host and guest, abstracting away the memory management complexities and enabling developers to write plugins in any supported language (Rust, Python, JavaScript, C, Go, etc.) without writing the laborious glue code themselves. The vision is to have a single, language-neutral interface definition (WIT - WebAssembly Interface Type) from which all necessary boilerplate code can be automatically generated, making the creation of multi-language, extensible systems significantly more maintainable and robust.

Key Findings

▶ Watch: What is WebAssembly (WASM) and its benefits (2:40)

The talk's primary findings revolve around the promise and current challenges of integrating the WebAssembly Component Model into the Kubernetes ecosystem, specifically for scheduler extensions.

1. The Component Model's Promise: The speakers confirmed that the WebAssembly Component Model offers a conceptually ideal solution for the existing problem of multi-language, maintainable Kubernetes WASM plugins. Its core value propositions – a canonical API for standardized data exchange, language-neutral interface definitions (WIT), and automated binding generation (bindgen) – directly address the issues of manual serialization, memory management, and the need for multiple, manually synchronized guest SDKs. This model would allow developers to define complex Kubernetes structures once in WIT and then automatically generate host-side Go code to call WASM components and guest-side SDKs for various languages (Rust, Python, Go, etc.) to implement the plugin logic.

1. Significant Tooling Immaturity: Despite its strong conceptual foundation, the Component Model is still in its early stages of development, particularly concerning tooling. A critical roadblock identified was the complete lack of WIT generators capable of converting existing, widely used schema definitions (like Kubernetes's Protobuf, OpenAPI, or native Go structs) into the required WIT format. This means that to use the Component Model, developers would currently need to manually rewrite vast, complex Kubernetes type definitions into WIT, an unmaintainable and prohibitive task for any real-world integration.

1. Lack of Go Runtime Support: Another major finding was the absence of mature Go runtime support for the Component Model. The existing Go WASM runtimes, Wazero (preferred for its zero-dependency nature) and Wasmtime-go, currently do not fully support the Component Model's features. While there are discussions and ongoing development, neither runtime is production-ready for Component Model hosts. This forced the speakers to explore alternatives and compromises, as their Kubernetes plugin framework is fundamentally written in Go.

1. Viable Alternatives (with compromises):

• Xism emerged as the "top contender alternative." While not a full Component Model implementation, it offers a pragmatic approach to multi-language WASM plugins. It supports many host and guest languages, including Go as a host and Rust, Python, JavaScript, C, and Go for guests. Xism simplifies the host-guest interface to a single input/output of bytes, abstracting away the memory management through its PDKs (Plug-in Development Kits). It enables the use of JSON or Protobuf for passing complex data, with the caveat that marshalling/unmarshalling logic must be explicitly handled on the Go host side and, in the case of Protobuf, currently also on the guest side if specific language-native types are desired.
• XTP Bindgen, built on Xism, aims to further automate glue code generation using an OpenAPI-like schema, but it crashed when attempting to convert Kubernetes schemas, indicating its own immaturity.
• Gravity, an early-stage transpiler building on WASI to implement the Component Model, showed promise in benchmarks but currently supports only a very limited set of primitive types, making it unsuitable for complex Kubernetes structures.

1. Performance Trade-offs and Benefits: Benchmarking revealed crucial performance characteristics:

• Using JSON for data exchange in Xism showed comparable performance across different guest languages (Rust, Python, JavaScript).
• Switching to VT proto (a faster Protobuf implementation) significantly improved performance over JSON when marshaling/unmarshalling on the Go host side.
• Early tests with Gravity (even for simple string passing) showed a slight performance edge over Xism's JSON method, hinting at the Component Model's potential.
• Critically, a Rust host using the WIT definition with a tiny Go guest (utilizing Wasmtime) demonstrated a substantial performance gain (much faster) compared to the same logic using JSON in Wasmtime, underscoring the efficiency benefits of the Component Model's canonical API.
• A comparison with a traditional CGO shared library call revealed that the overhead of WASM (specifically Xism with JSON) is in a similar ballpark to CGO with JSON serialization, reinforcing that WASM is a viable, safe alternative to risky shared memory approaches.

In summary, while the WebAssembly Component Model represents the future of extensible systems like Kubernetes, its current ecosystem lacks the maturity in tooling (specifically WIT generators) and Go runtime support to be immediately practical for complex, real-world applications. Pragmatic alternatives like Xism offer a workable compromise, but the full benefits of the Component Model, particularly in performance and developer experience, are yet to be fully realized and require significant community effort.

Technical Deep Dive

▶ Watch: Kubernetes WASM Plugin Architecture with Wazero (3:30)

The technical journey presented in the talk dissects the evolution from existing, cumbersome WASM integration patterns to the ambitious, yet currently challenging, adoption of the WebAssembly Component Model.

Current WASM Plugin Mechanism in Kubernetes

Jonathan detailed the current state of WASM plugins within the Kubernetes scheduler extension. A host-side Go function, like Filter, receives a Pod and NodeInfo. Before calling into the WASM guest, a "pref-filter" serializes the Pod into Protobuf and places it into a memory buffer. The actual call to the WASM module is then mediated by the Wazero runtime.

On the guest side (inside the WASM VM), the user's plugin code (e.g., a filter function) doesn't directly receive complex types. Instead, it receives pointers and integers from the stack. The guest SDK, which is auto-generated but currently only for "tiny Go", performs "memory shenanigans." For instance, to retrieve a string like the node name, the guest SDK calls back into the host via an imported function (mem.get_string). This host function then writes the string into a buffer within the guest's memory, which the guest can then access. This bidirectional communication for complex types, involving manual buffer management and serialization/deserialization, is the core complexity that the Component Model aims to eliminate.

The WebAssembly Component Model: A Vision

Dejan introduced the WebAssembly Component Model as the proposed solution. Its key features are:

• Standardized, Language-Neutral Interfaces: The Component Model uses WIT (WebAssembly Interface Type) files to define clear interfaces and complex data structures (like Kubernetes Pod or NodeInfo) in a language-agnostic manner. This is a crucial departure from the current setup where these structures are implicitly defined by Go types and Protobuf schemas.
• Canonical API: This low-level binary interface dictates exactly how complex data types are represented, serialized, and how references are managed when passed across language boundaries. This standardization is fundamental to achieving seamless interoperability.
• Automatic Binding Generation: With a WIT definition, tools like bindgen (for Rust) can automatically generate host-side code (e.g., Rust code to instantiate a Pod and call a WASM filter) and guest-side SDKs for various languages. This means a developer would only write the core logic of their plugin, and all the "glue" code for interacting with the host and handling complex types would be generated.

The speakers demonstrated how a WIT file could define a FilterInput containing Pod and NodeInfo structs, and an Export method for the filter function. This WIT definition would then be used to generate robust, type-safe bindings, eliminating the manual memory management seen in the current Go-only WASM SDK.

Roadblocks and Alternatives Explored

The ambitious plan to refactor the Kubernetes WASM extension with the Component Model hit two major roadblocks:

1. Lack of WIT Generators: The most significant issue was the absence of tools to automatically generate WIT files from existing Kubernetes schemas (Protobuf, OpenAPI, or Go structs). Manually rewriting the extensive Kubernetes object definitions into WIT was deemed unfeasible and unmaintainable.
2. No Go Host Runtime for Component Model: The existing Go WASM runtimes, Wazero and Wasmtime-go, lacked full Component Model support. This was a critical blocker since the Kubernetes plugin framework is written in Go.

Given these limitations, Jonathan explored several alternatives:

• Xism: This framework emerged as the most viable alternative. Xism takes a simpler approach, standardizing the host-guest interface to a single input and output of bytes. It provides PDKs (Plug-in Development Kits) for many languages (Rust, JavaScript, Python, C, Go), which abstract away the memory "shenanigans" for the guest developer.
• JSON with Xism: Xism allows passing JSON strings. A JSON schema can define the FilterInput (containing Pod and NodeInfo). On the Go host, manual JSON marshalling/unmarshalling is required. On the guest side (e.g., Rust using its PDK), Xism provides plumbing to receive and return native complex types, simplifying the guest developer's experience.
• Protobuf with Xism: For performance-critical scenarios, Protobuf can be used. On the Go host, using VT proto (a highly optimized Protobuf implementation) for marshalling/unmarshalling significantly boosts performance. However, on the guest side, the current Xism PDKs don't fully abstract Protobuf types as elegantly as JSON, often requiring manual unmarshalling from bytes. This presents a trade-off between performance and developer convenience.
• XTP Bindgen: This tool, built on Xism, aims for even more automation by defining a schema similar to OpenAPI for function signatures and data types. It promises to generate even more of the glue code. However, initial attempts to convert Kubernetes OpenAPI schemas failed, indicating its early stage.
• Gravity: An experimental project released in January, Gravity is a transpiler built on WASI to implement the Component Model. While it's very early and supports only primitive types (strings, integers), it represents a promising direction for native Component Model support in Go.

Performance Benchmarks

Jonathan presented benchmark results comparing the various approaches for a simple "regex filter" plugin that checks a pod annotation against a node name.

• Xism (Go Host, JSON):
• Rust guest: ~30,000 ops/sec
• Python guest: ~20,000 ops/sec
• JavaScript guest: ~15,000 ops/sec
• These showed consistent performance across different guest languages, indicating the overhead of JSON serialization.
• Xism (Go Host, VT Proto):
• With VT proto, the performance significantly improved to ~80,000 ops/sec, demonstrating the benefit of efficient serialization.
• Gravity (Go Host, JSON string):
• Even for simple string passing, Gravity achieved ~35,000 ops/sec, slightly faster than Xism's JSON, suggesting potential inherent efficiency advantages.
• Wasmtime (Rust Host, WIT):
• Using a Rust host with a small subset of the Component Model's WIT and a tiny Go guest, the performance jumped to ~120,000 ops/sec. This was significantly faster than using JSON with Wasmtime (~70,000 ops/sec) for the same code, highlighting the substantial performance gains from the canonical API and direct type passing.
• CGO (Go Host, JSON):
• For comparison, a Go host calling a shared library written in Go via CGO (using JSON serialization) achieved ~25,000 ops/sec. This showed that WASM with JSON is in a similar performance ballpark to CGO with JSON, but WASM offers sandboxing and safer memory management, avoiding the dangers of shared memory in CGO.

The benchmarks strongly suggest that while Xism offers a pragmatic solution with acceptable performance, the full WebAssembly Component Model with its WIT and canonical API holds the key to superior performance by eliminating serialization overhead and enabling direct, type-safe data exchange.

Demo / Proof of Concept

▶ Watch: Current limitation: WASM plugins only support TinyGo (4:40)

While the talk did not feature a live, interactive demonstration, the speakers consistently referenced and used a specific example plugin as their proof of concept throughout their technical explanations and benchmarks. This example was a "filter plug-in" designed for the Kubernetes scheduler.

The logic of this proof of concept plugin was described as follows:

The plugin reads an annotation from the Pod definition. This annotation is expected to contain a regular expression. The plugin then compiles this regular expression. Subsequently, it applies the compiled regex to the Node name where the pod is being considered for scheduling. Based on whether the node name matches the regex, the plugin makes a decision to either filter out (prevent scheduling on) or allow the pod to be scheduled on that specific node.

This simple yet practical filter was chosen because it involves:

1. Receiving complex inputs (Pod and NodeInfo).
2. Performing some computational logic (regex compilation and matching).
3. Returning a status (allow or deny).

By implementing this specific filter plugin using various approaches—the original Go-only WASM SDK, Xism with JSON, Xism with Protobuf, and finally, a Rust host with the Component Model's WIT using a tiny Go guest—the speakers were able to effectively compare the developer experience, the complexity of host-guest interaction, and the performance characteristics of each method. This allowed them to illustrate the "memory shenanigans" of the traditional approach, the compromises of Xism, and the potential efficiency gains of the nascent Component Model.

Defensive Implications

▶ Watch: Deep dive into complex host-guest memory interaction (6:50)

While this talk primarily focuses on the extensibility and performance of the Kubernetes scheduler rather than direct security vulnerabilities, the adoption of WebAssembly and particularly the Component Model carries significant defensive implications for cloud-native environments.

1. Enhanced Sandboxing and Isolation: WebAssembly inherently provides a strong sandboxing model. Unlike traditional native plugins (like shared libraries loaded via CGO), WASM modules execute in a secure, isolated environment, preventing them from directly accessing the host system's memory or resources without explicit permissions. This significantly reduces the attack surface for custom scheduler logic. If a malicious or buggy plugin were to be deployed, its impact would be contained within its WASM sandbox, preventing arbitrary code execution or privilege escalation on the Kubernetes control plane. This is a crucial defensive advantage over less isolated extension mechanisms.

1. Improved Supply Chain Security: The Component Model aims to standardize interfaces and enable automatic generation of language bindings. This reduces the amount of manually written glue code, which is often a source of subtle bugs and potential vulnerabilities. By relying on well-tested, automatically generated code, the risk of introducing flaws through complex serialization, deserialization, or memory management is minimized. Furthermore, the clear interface definitions in WIT files can facilitate better auditing and understanding of how data flows between components, enhancing the transparency of the software supply chain for these extensions.

1. Increased Maintainability and Reduced Error Surface: The ability to write scheduler plugins in multiple languages using automatically generated SDKs drastically improves maintainability. Developers can use their preferred, familiar tools and languages, leading to fewer errors and more robust code. A codebase that is easier to understand and maintain is inherently more secure, as bugs (including security bugs) are less likely to be introduced and easier to identify and fix. The current "memory shenanigans" required for complex type passing are a prime example of error-prone code that the Component Model seeks to eliminate, thus reducing a common source of vulnerabilities.

1. Performance and Resilience: While not directly a security feature, the performance improvements demonstrated by the Component Model (especially with WIT and VT proto) contribute to the overall resilience of the Kubernetes control plane. A more efficient scheduler can handle higher loads, process scheduling decisions faster, and operate more reliably, even with complex custom policies. A performant and stable control plane is less susceptible to denial-of-service conditions or operational failures that could indirectly create security weaknesses.

1. Standardization and Interoperability: The Component Model's emphasis on standardization through the canonical API and WIT promotes better interoperability between different components and across different languages. This standardization can lead to a more coherent and predictable ecosystem for Kubernetes extensions, making it easier to integrate, test, and secure custom logic without encountering unexpected incompatibilities or security gaps stemming from ad-hoc integration methods.

In essence, while the talk presented a developer experience and performance narrative, the underlying technologies, particularly WebAssembly and the Component Model, offer a robust foundation for building more secure, resilient, and auditable custom logic within critical infrastructure components like the Kubernetes scheduler.

Key Takeaways

• The WebAssembly Component Model is the future for extensible systems like Kubernetes: It offers a standardized, language-neutral way to define interfaces and complex types, promising automatic generation of host and guest SDKs across many programming languages.
• Current Kubernetes WASM plugins are limited by Go-only SDKs and complex manual memory management: Existing approaches require intricate serialization and pointer handling, making multi-language support and maintenance a significant challenge.
• Component Model tooling and Go runtime support are still immature: A major roadblock is the lack of WIT generators for existing schema formats (Protobuf, OpenAPI, Go structs) and the absence of full Component Model support in Go runtimes like Wazero and Wasmtime-go.
• Xism offers a pragmatic, albeit compromised, alternative for multi-language WASM plugins: It provides PDKs that abstract memory management, allowing complex types to be passed via JSON or Protobuf, offering a workable solution today with performance trade-offs.
• The Component Model's canonical API delivers significant performance benefits: Benchmarks show that direct type passing via WIT (with Wasmtime) can be substantially faster than JSON serialization, making it ideal for performance-critical scenarios.
• Community collaboration is crucial for advancing the Component Model: The speakers issued a call to action for contributors to develop WIT generators, improve Go runtime support, and generally accelerate the Component Model's maturity to unlock its potential across the CNCF landscape.

About the Speaker(s)

Dejan Pejchev is an Open Source Engineer with the G Research open source team. He is passionate about applying new technologies like the WebAssembly Component Model to complex systems, as demonstrated by his work on the Kubernetes scheduler. Dejan brings expertise in navigating the challenges of integrating cutting-edge solutions into existing infrastructure.

Jonathan Giannuzzi also works as an Open Source Engineer at G Research. He focuses on open source projects across various programming languages, always with a keen eye on performance and maintainability. Jonathan has a deep understanding of the Kubernetes scheduler and its extensibility, having previously presented on the topic at KubeCon. His work involves extensive research into optimizing host-guest interactions and exploring alternative runtime environments for WebAssembly.