Project Lightning Talk: Maturing Bare Metal Provisioning With Metal3 - Ádám Rozmán, Maintainer

Project Lightning Talk: Maturing Bare Metal Provisioning With Metal3

Speakers: Ádám Rozmán; Maintainer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=cs68TjSAlTg

Overview

In this lightning talk at KubeCon EU, Ádám Rozmán, a maintainer of the Metal3 project, provided a comprehensive update on the project's journey towards maturity and its significant advancements in enabling Kubernetes deployments on bare metal infrastructure. Metal3 stands as a crucial component in the cloud-native ecosystem, offering an end-to-end, full-stack solution for provisioning and managing Kubernetes clusters directly on physical machines. It eliminates the need for virtualized environments or traditional cloud providers, allowing users to leverage the full performance and control of bare metal hardware.

The project's core strength lies in its integration with Cluster API (CAPI), a Kubernetes sub-project that brings declarative, Kubernetes-native cluster lifecycle management to various infrastructure providers. Metal3 extends CAPI's capabilities to the bare metal domain, transforming physical servers into CAPI-managed resources. This talk highlighted the substantial growth in features, community engagement, and security enhancements that underscore Metal3's increasing stability and readiness for broader enterprise adoption. The progress towards CNCF incubation status further validates its importance and sustained development.

Metal3 addresses a critical need for organizations seeking to deploy Kubernetes in environments where bare metal is preferred or required, such as high-performance computing (HPC), edge computing, or private cloud scenarios with stringent resource isolation and cost efficiency demands. Its adoption by major enterprises like Ericsson, Red Hat, SUSE, and Fujitsu demonstrates its practical utility and robustness in complex production environments. Rozmán's presentation served as a testament to the project's vibrant community and its commitment to delivering a robust, secure, and feature-rich solution for bare metal Kubernetes.

Background

▶ Watch: Introduction to Metal3: Kubernetes on bare metal (0:00)

The journey to deploying and managing Kubernetes clusters on bare metal machines has historically been fraught with complexity. Unlike public cloud environments, where infrastructure provisioning is abstracted and automated by cloud providers, bare metal requires direct interaction with physical hardware, including network booting, operating system installation, and hardware-specific configurations. This manual, often script-driven process is time-consuming, error-prone, and lacks the declarative, API-driven elegance that Kubernetes users have come to expect.

This challenge led to the inception of projects like Metal3. Established in 2019, Metal3 emerged to bridge the gap between Kubernetes' declarative management paradigm and the intricacies of bare metal infrastructure. Its fundamental approach involves treating bare metal machines as Kubernetes resources, allowing users to define their desired state using standard Kubernetes YAML manifests. This is achieved by integrating with the Cluster API (CAPI), an ambitious project that aims to bring declarative provisioning and lifecycle management to Kubernetes clusters themselves. CAPI defines a set of Custom Resource Definitions (CRDs) like Cluster, Machine, and MachineDeployment, which abstract away the underlying infrastructure details. Metal3 functions as a Cluster API infrastructure provider for bare metal, implementing the necessary controllers to translate CAPI's generic machine requests into concrete actions on physical servers.

The project's maturity journey has been marked by several significant milestones. After its establishment, Metal3 joined the CNCF Sandbox in September 2020, signifying its recognition within the broader cloud-native community. The project has since demonstrated sustained development and community growth, leading to an application for CNCF Incubation in December 2023. This application process, which included rewriting the incubation proposal to align with evolving CNCF requirements and undergoing a thorough due diligence review, including adopter interviews, highlights the project's commitment to open governance, stability, and widespread usability. Metal3 aims to provide a reliable, automated, and Kubernetes-native method for provisioning and managing bare metal infrastructure, thereby extending the power of cloud-native principles to the physical layer.

Key Findings

▶ Watch: Project growth, releases, and community contributions (1:21)

Ádám Rozmán's talk underscored the substantial progress and maturation of the Metal3 project, presenting a compelling narrative of growth, feature expansion, and a strong commitment to security and community. The project's evolution since its inception in 2019 and particularly since KubeCon North America last November reveals a robust and active development cycle.

Quantitatively, Metal3 has seen impressive growth metrics:

• Feature Development: Approximately five new major features have been introduced since KubeCon NA, contributing to a total of 65-70 major features since the project joined the CNCF Sandbox. This consistent delivery of new capabilities demonstrates a dynamic development pipeline.
• Release Cadence: The project has issued 34 releases across all its components combined since last November, indicating a regular and reliable release schedule that brings new functionalities and fixes to users promptly.
• Code Contributions: Over 440 different commits have been merged since November, and a remarkable 5,500 commits since joining the sandbox. These figures highlight a highly active contributor base and continuous code refinement.
• Community Expansion: The project welcomed one new adopter since November, adding to its growing list of enterprise users such as Ericsson, Red Hat, SUSE, and Fujitsu. The number of dedicated maintainers remains stable at 16, ensuring consistent project stewardship.
• Community Engagement: GitHub statistics reveal approximately 10,000 different GitHub events—including reviews, mergers, comments, and commits—since last November, reflecting a vibrant and engaged community actively participating in the project's evolution.

Beyond these metrics, the talk emphasized key achievements in the project's maturity and strategic direction:

• CNCF Incubation Progress: Metal3 has successfully navigated critical stages of the CNCF incubation process, including passing adopter interviews and completing the due diligence review. While awaiting the final results, this significant progress indicates the project's readiness for a higher level of CNCF endorsement, which typically signifies broader adoption potential and long-term sustainability.
• Security Focus: A dedicated security team has been instrumental in handling "three major security issues," demonstrating a proactive and responsible approach to vulnerability management and overall project hardening. This commitment is crucial for enterprise-grade solutions.
• Advanced Feature Introductions: The project is actively integrating several cutting-edge features:
• Online database migration for cluster nodes, enhancing operational flexibility and minimizing downtime during upgrades or reconfigurations.
• Read-only file system support for one of its components, bolstering security by preventing unauthorized modifications to critical system files.
• Adaptation to support the Cluster API multi-tenancy architecture, allowing for more secure and efficient sharing of bare metal infrastructure among multiple teams or users.
• Introduction of support for OCI disk images, enabling the distribution and management of full operating system images using the Open Container Initiative (OCI) standard, traditionally used for container images. This brings standardization and immutability benefits to OS provisioning.
• Early adoption of autonomous disk encryption, providing robust data protection for bare metal machines without manual intervention, crucial for compliance and data privacy.

These findings collectively paint a picture of a project that is not only growing rapidly in terms of features and code but is also maturing strategically, addressing critical enterprise requirements for security, operational efficiency, and scalability within the bare metal Kubernetes ecosystem.

Technical Deep Dive

▶ Watch: Metal3's journey towards CNCF incubation status (2:40)

Metal3's technical architecture is fundamentally built around its role as a Cluster API infrastructure provider for bare metal environments. This means it extends the Kubernetes API to manage physical hardware, allowing users to provision and lifecycle-manage bare metal machines and the Kubernetes clusters running on them using standard Kubernetes tools and declarative manifests.

At its core, Metal3 introduces several Custom Resource Definitions (CRDs) that represent bare metal resources within Kubernetes. The most central of these is the BareMetalHost CRD, which describes a physical server, including its hardware details (e.g., MAC address, IPMI/Redfish credentials for out-of-band management), network configuration, and desired operating system image. Metal3 controllers watch for changes to these BareMetalHost objects and orchestrate the provisioning process.

The typical workflow orchestrated by Metal3 involves several stages:

1. Discovery: Metal3 can discover available bare metal machines in the network, often leveraging technologies like PXE boot or Redfish/IPMI for hardware introspection.
2. Provisioning Request: A user or a Cluster API Machine controller creates a BareMetalHost resource, specifying the desired OS image and other configurations.
3. OS Deployment: The Metal3 controller uses the BareMetalHost information to initiate a network boot (PXE or iPXE) for the physical machine. It then deploys the specified operating system image onto the machine's local storage. This image can be a standard ISO or, as highlighted in the talk, an OCI disk image.
4. Configuration: Once the OS is installed, Metal3 can inject initial configuration (e.g., SSH keys, network settings, cloud-init scripts) to prepare the machine for Kubernetes.
5. Kubernetes Cluster Installation: The CAPI Machine controller, in conjunction with a bootstrap provider (e.g., Kubeadm), then takes over to install Kubernetes components (kubelet, kube-proxy, etc.) and join the machine to the cluster.

The talk highlighted several key technical advancements that enhance Metal3's capabilities:

• Online Database Migration for Cluster Nodes: This feature is crucial for maintaining high availability and operational flexibility in Kubernetes clusters. For core components like etcd, which serves as the cluster's database, performing migrations or upgrades traditionally requires careful planning to minimize downtime. Online migration capabilities imply that database schema changes or data relocations can occur while the cluster remains operational, significantly improving the maintainability and resilience of bare metal Kubernetes deployments. This is especially relevant for etcd clusters, where data consistency and availability are paramount.

• Read-Only File System Support: While the specific component benefiting from this was not named, implementing a read-only file system for critical components enhances the security posture and stability of the Metal3 control plane or managed nodes. By preventing unauthorized writes to essential directories, it reduces the attack surface, makes it harder for malware to persist, and simplifies recovery from configuration drift or malicious tampering. This aligns with the principle of immutable infrastructure.

• Cluster API Multi-Tenancy Architecture Support: As organizations grow, the ability to share infrastructure securely among different teams or projects (tenants) becomes vital. CAPI is evolving to support multi-tenancy, and Metal3's adaptation ensures that bare metal resources can be allocated and isolated effectively for different tenants within a single Kubernetes management cluster. This typically involves robust Role-Based Access Control (RBAC), network policies, and potentially resource quotas to ensure that tenants can only access and manage their designated bare metal machines and clusters, preventing cross-tenant interference and enhancing security.

• OCI Disk Image Support: This is a significant innovation. Traditionally, operating system images for bare metal provisioning are distributed as ISO files, disk images (e.g., raw, qcow2), or through custom mechanisms. By supporting OCI disk images, Metal3 leverages the standards developed for container images. This means OS images can be stored, distributed, and managed in container registries (like Docker Hub or Harbor), benefiting from features such as content addressability, cryptographic signing, and efficient layer-based storage. This standardization streamlines the OS provisioning pipeline, improves image integrity, and facilitates better supply chain security for the foundational operating systems.

• Autonomous Disk Encryption: Data security at rest is a critical concern, especially in bare metal environments where physical access might be a risk. Autonomous disk encryption implies that the physical machines provisioned by Metal3 can have their disks encrypted automatically without manual intervention. This often involves integrating with hardware security modules (HSMs) like Trusted Platform Modules (TPMs), leveraging technologies like LUKS (Linux Unified Key Setup), and securely managing encryption keys, potentially integrating with Kubernetes secrets or external key management systems. This feature provides a robust layer of data protection, ensuring that even if a physical machine is compromised or stolen, the data on its disks remains inaccessible without the appropriate decryption keys.

These technical advancements demonstrate Metal3's continuous evolution towards a more resilient, secure, and operationally efficient platform for managing bare metal infrastructure within a Kubernetes-native framework. By embracing these sophisticated features, Metal3 empowers users to build highly secure and manageable bare metal cloud-native environments.

Demo / Proof of Concept

▶ Watch: Key new features: multi-tenancy, OCI images, encryption (3:40)

The lightning talk delivered by Ádám Rozmán at KubeCon EU focused primarily on providing an update on the Metal3 project's maturity, growth, and upcoming features. Due to the rapid-fire nature of a lightning talk, a live demonstration or detailed proof of concept was not presented during the session itself. The speaker instead highlighted the project's presence at a kiosk during the event, encouraging attendees to visit for further discussions, potential live interactions, and to engage with maintainers and contributors.

While a specific demo was not part of this particular presentation, the Metal3 project, as an active open-source initiative, provides extensive documentation, tutorials, and community resources for users interested in exploring its capabilities. These resources typically include guides for setting up a Metal3-managed bare metal environment, deploying Kubernetes clusters, and demonstrating the lifecycle management features that were discussed at a high level during the talk. Users can find more information through the project's official channels, including its GitHub repositories, documentation portals, and community Slack channels, to experience the project's functionalities firsthand.

Defensive Implications

▶ Watch: How to join and contribute to the Metal3 community (4:40)

The advancements and strategic focus areas presented for the Metal3 project carry significant defensive implications for organizations deploying Kubernetes on bare metal. A robust bare metal provisioning solution is foundational to the security posture of the entire cloud-native stack.

1. Enhanced Supply Chain Security through OCI Disk Images: The adoption of OCI disk images for operating system provisioning is a crucial step in strengthening the software supply chain. By utilizing the OCI standard, organizations can leverage existing tooling for content addressability, cryptographic signing, and vulnerability scanning, similar to how container images are secured. This ensures that the base operating system deployed on bare metal machines is authentic, untampered, and free from known vulnerabilities, significantly reducing the risk of supply chain attacks targeting the OS layer. Defenders can integrate OCI image registries with security scanners and policy enforcement tools to validate OS images before deployment.

1. Data Protection with Autonomous Disk Encryption: The introduction of autonomous disk encryption directly addresses a critical security requirement: data at rest protection. In bare metal environments, where physical access to servers is a tangible risk, full disk encryption (FDE) is paramount. Automating this process ensures consistent encryption across all provisioned machines, eliminating manual misconfigurations or forgotten steps. This feature, likely leveraging technologies like LUKS and potentially TPMs for key management, safeguards sensitive data from unauthorized access even if physical hardware is compromised or stolen. Defenders should ensure proper key management strategies are in place, integrating with existing KMS solutions where possible.

1. Reduced Attack Surface with Read-Only File Systems: Implementing read-only file system support for critical components is a proactive security measure. By restricting write access to essential system directories, it significantly reduces the attack surface and mitigates the impact of potential compromises. Malicious actors would find it much harder to establish persistence, modify system binaries, or inject malicious code into protected areas. This aligns with the principle of immutable infrastructure, where changes are made by deploying new, verified images rather than modifying running systems.

1. Secure Multi-Tenancy for Isolation: Adapting to the Cluster API multi-tenancy architecture is vital for enterprises that need to share bare metal resources among multiple teams or business units. Proper multi-tenancy ensures strong isolation between different tenants' clusters and data. This prevents one compromised tenant from affecting others and allows for granular access control. Defenders must ensure that the multi-tenancy implementation enforces strict network segmentation, resource quotas, and RBAC policies to maintain this isolation effectively.

1. Proactive Security Team and Vulnerability Management: The explicit mention of a dedicated security team handling "three major security issues" highlights a mature approach to security. This indicates that Metal3 is actively identifying, triaging, and remediating vulnerabilities, which is crucial for maintaining trust and stability. Defenders relying on Metal3 can have greater confidence in the project's commitment to security and its ability to respond effectively to emerging threats. Regularly checking the project's security advisories and release notes is recommended.

1. OpenSSF Badge and Best Practices: While not explicitly detailed, the mention of checking the project's OpenSSF badge implies adherence to industry-recognized security best practices. The OpenSSF (Open Source Security Foundation) provides frameworks and badges to assess and improve the security posture of open-source projects. This commitment to security hygiene helps ensure that Metal3 is developed with security in mind from the ground up, providing a more resilient foundation for bare metal Kubernetes deployments.

In summary, Metal3's latest advancements provide a more secure and resilient foundation for bare metal Kubernetes clusters. Defenders should leverage these features to harden their infrastructure, streamline security operations, and reduce the overall risk profile of their cloud-native environments.

Key Takeaways

• Maturing Bare Metal Provisioning: Metal3 offers a robust, end-to-end solution for provisioning and managing Kubernetes clusters on bare metal machines, providing a Kubernetes-native experience for physical infrastructure.
• Significant Project Growth: The project demonstrates strong momentum with numerous new features, a high volume of code commits, frequent releases, and a growing adopter base, reflecting its increasing stability and community engagement.
• Progress Towards CNCF Incubation: Metal3 has successfully completed critical stages of the CNCF incubation review, including adopter interviews and due diligence, signaling its readiness for broader industry adoption and long-term sustainability.
• Advanced Feature Set: Key new capabilities include online database migration for cluster nodes, read-only file system support for enhanced security, adaptation for Cluster API multi-tenancy, and critical bare metal security features like OCI disk image support and autonomous disk encryption.
• Strong Security Posture: A dedicated security team actively addresses vulnerabilities, and features like autonomous disk encryption and read-only file systems significantly bolster the security of bare metal Kubernetes deployments.
• Cluster API Integration: As a Cluster API infrastructure provider, Metal3 seamlessly integrates with the broader Kubernetes ecosystem, enabling declarative management of bare metal infrastructure alongside virtualized or cloud-based environments.

About the Speaker(s)

Ádám Rozmán is identified as a Maintainer of the Metal3 project. In this capacity, he plays a crucial role in guiding the project's technical direction, reviewing contributions, and ensuring the overall health and progress of the Metal3 ecosystem. His involvement as a maintainer underscores his deep technical expertise and commitment to advancing bare metal provisioning within the cloud-native landscape. No further biographical details were provided in the transcript.