Cloud Native Wheel of Fortune: 5 Spins for 5 Topics! - Steve Wade, & Matteo Bianchi

Cloud Native Wheel of Fortune: 5 Spins for 5 Topics!

Speakers: Steve Wade, Cloud Native Catalyst; Matteo Bianchi, CNCF Ambassador, Solution Engineer at GitHub

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Pmba7R4_4oU

Overview

In a refreshing departure from traditional conference formats, Steve Wade and Matteo Bianchi presented "Cloud Native Wheel of Fortune," an interactive session at KubeCon EU designed to address the audience's most pressing questions. Instead of a pre-determined agenda, the speakers engaged attendees through a live "wheel of fortune" spin, allowing the audience to collectively choose five key cloud-native topics for discussion. This innovative approach transformed a standard talk into a dynamic, "speed dating for cloud native" experience, delivering essential insights on diverse subjects ranging from air-gapped environments and eBPF to scalability, sustainability, and GitOps.

The talk’s core premise was to provide essential foundational elements across the vast and complex cloud-native landscape, emphasizing that successful cloud-native adoption is less about chasing every new shiny technology and more about strategically selecting and integrating tools that align with specific organizational challenges. Wade and Bianchi, drawing from their extensive experience in platform engineering and cloud-native advocacy, showcased the interconnectedness of these seemingly disparate topics. They underscored the importance of community engagement, continuous learning, and focusing on business outcomes rather than merely technology adoption.

This article delves into the five topics explored during the session, providing a technical deep dive into the problems, solutions, and practical implications discussed. It highlights the speakers' insights, real-world examples, and actionable advice for organizations navigating the intricacies of modern cloud-native infrastructures. The interactive format itself served as a testament to the community-driven spirit of KubeCon, allowing the content to directly reflect the immediate interests and concerns of the attendees.

Background

▶ Watch: Introduction to Cloud Native Wheel of Fortune format (0:00)

The rapid evolution and increasing complexity of the cloud-native ecosystem have presented organizations with a plethora of choices and challenges. As applications become more distributed and infrastructure more ephemeral, critical areas like security, observability, scalability, and operational efficiency demand innovative solutions. The traditional "one-size-fits-all" approach to infrastructure management is no longer viable, leading to a constant search for best practices and effective tooling. This talk aimed to cut through some of that complexity by directly engaging the audience on topics they deemed most relevant.

The interactive "Wheel of Fortune" format was conceived to directly address a common frustration among conference-goers: the desire for speakers to tackle specific questions relevant to their individual challenges. By allowing the audience to vote on topics, Wade and Bianchi ensured that the discussions were immediately pertinent. The chosen topics—air-gapped environments, eBPF, scalability, sustainability, and GitOps—represent significant, often challenging, facets of cloud-native adoption that many organizations grapple with daily. These areas are frequently intertwined, and a deep understanding of each is crucial for building resilient, secure, and efficient cloud-native platforms. The speakers' collective background in platform engineering, large-scale Kubernetes deployments, and automation provided a strong foundation for addressing these diverse and critical subjects.

Key Findings

▶ Watch: Audience chooses 'Air-gapped Environments' as first topic (2:00)

The talk, through its interactive format, revealed several key findings across the five chosen cloud-native topics:

• Air-gapped Environments: Running Kubernetes in fully air-gapped or semi-air-gapped environments presents unique and often complex challenges, particularly concerning image registry deployment and ensuring the provenance and attestation of container images. Networking is identified as the most difficult aspect, alongside resource constraints common in edge deployments. The speakers highlighted the growing importance of SBOMs (Software Bill of Materials) for secure image management, even for bootstrapping clusters.
• eBPF: This technology is revolutionizing how we secure, monitor, and optimize systems at scale by transforming the kernel into a programmable interface. It allows for advanced kernel-level operations from user space without needing to patch the kernel, providing a sandboxed environment for programs. eBPF is a fundamental building block for future observability, security, and networking projects, with tools like Falco demonstrating significant impact on reducing security incidents.
• Scalability: Successfully scaling Kubernetes goes beyond merely adding more nodes. It requires a holistic approach addressing control plane performance, simplifying the developer experience for large teams, managing geographical distribution, and implementing robust multi-tenancy models. Solutions often involve a combination of physical and virtual clusters, along with sophisticated policy enforcement mechanisms like service meshes.
• Sustainability: Cloud-native adoption has a significant environmental impact, necessitating a shift towards carbon-aware computing. This involves prioritizing carbon intensity, energy efficiency, and measuring impact, sometimes even at the expense of marginal cost savings. Practices like optimizing code, leveraging efficient containerization, and choosing green cloud regions contribute significantly to reducing carbon footprints.
• GitOps: This methodology establishes Git as the single source of truth for declarative infrastructure and application configurations, fundamentally shifting operational mindsets. It simplifies multi-cluster consistency, enhances auditability, and streamlines compliance for Kubernetes deployments and beyond, making manual interventions less frequent and more controlled.

A cross-cutting finding emphasized by the speakers is the interconnectedness of these cloud-native topics. Solutions in one area often impact others, and true success comes from strategically integrating tools that solve specific challenges, rather than indiscriminately adopting every new technology. The most successful organizations, as noted, build a strategic foundation first and then evolve based on actual needs.

Technical Deep Dive

▶ Watch: Challenges of air-gapped Kubernetes deployments (3:00)

Air-gapped Environments

Operating Kubernetes in air-gapped environments poses substantial technical hurdles. Steve Wade recounted experiences in highly regulated financial institutions where even USB sticks were forbidden. The primary challenge isn't merely bootstrapping Kubernetes—a relatively well-solved problem—but rather securely managing and distributing the necessary container images. Deploying an image registry within such an environment, one that can be leveraged to bootstrap all Kubernetes clusters, proved to be a far more complex task than anticipated. This involves not only the registry itself but also ensuring the attestation and provenance of all images, including those foundational to Kubernetes. The speakers highlighted the increasing availability of SBOMs (Software Bill of Materials) in recent Kubernetes releases as a crucial step towards addressing this.

Matteo Bianchi added that networking is often the most difficult part of air-gapped setups, particularly in semi-air-gapped scenarios where data ingress is allowed but egress is restricted. Furthermore, resource constraints are prevalent in edge deployments, such as vehicles, satellites, or other remote devices, where limited compute, memory, and high network latency prevent leveraging the "goodies" of typical cloud environments. Solutions often involve a mix of "being savvy" with networking, utilizing specialized, lightweight, and immutable operating systems like Tails, and careful capacity planning, acknowledging that elasticity is limited to the confines of the local data center.

eBPF

eBPF (extended Berkeley Packet Filter) was presented as a revolutionary technology for securing, monitoring, and optimizing systems at scale. Matteo Bianchi, known for his "eBPF to my grandma" talks, explained that eBPF transforms the Linux kernel into a programmable interface. This allows developers to attach sandboxed programs to various kernel events without modifying the kernel source code or loading kernel modules. This user-space interaction with kernel capabilities drives significant innovation in observability (tracing, profiling), advanced networking (e.g., Cilium for CNI and load balancing), and deep security applications (e.g., Falco for runtime security).

Developing eBPF programs traditionally requires C, which can be a barrier for many. However, libraries in languages like Go are making eBPF more accessible. Bianchi shared a real-world example from a large fashion company where their multi-cloud infrastructure was "more open than a brew pub on a Friday night," leading to a crypto miner running unnoticed and costing millions in infrastructure. Under Bianchi's advisory, they adopted Falco, an eBPF-based runtime security tool, to replace legacy and non-existent security solutions. By deploying the Falco agent across VMs and Kubernetes clusters and implementing custom policies, they achieved a 25% reduction in security incidents. Integration with tools like Sidekick and real-time messaging on Slack or Teams further enhanced their security posture. While not a "holy grail," eBPF tools like Falco provide fundamental building blocks for robust security, monitoring, and observability.

Scalability

Scaling Kubernetes to thousands of workloads and tens of thousands of nodes introduces complex challenges that go beyond simple resource provisioning. Steve Wade outlined four key areas for successful scaling: Nodes, Teams, Regions, and Multi-tenancy.

• Nodes: Focus on maintaining control plane performance and consider architectures like virtual clusters (e.g., using vcluster) or federated clusters to distribute the load.
• Teams: Simplify the onboarding experience and ensure strong isolation using namespace isolation, resource quotas, and network policies.
• Regions: Address geographical distribution with federated clusters spanning multiple regions or dedicated clusters in different regions.
• Multi-tenancy: Evaluate options like virtual clusters (providing dedicated control planes on a shared underlying cluster) or entirely dedicated physical clusters for different tenants, finding a balance between management overhead and isolation.

Wade cited an example from a media streaming platform in the Middle East, operating over 50 clusters with approximately 10,000 nodes and serving 150 product teams. Initially, they hit control plane limits trying to run everything on a single "supercluster." Their solution involved a hybrid approach combining physical clusters managed by Cluster API with virtual clusters using vcluster. They enforced policies across clusters using custom admission controllers and, crucially, adopted a service mesh to provide scalable and easily leveraged network policies, simplifying a previously "very, very complex network policy configuration." This strategy allowed developers to perceive a single "super cluster" while the underlying complexity was managed, leading to increased platform availability and a substantial decrease in the need for developers to understand intricate infrastructure details. For network congestion, the strategy involved having dedicated clusters for specific functions (e.g., function A on one cluster, function B on another) to minimize inter-cluster communication.

Sustainability

Matteo Bianchi highlighted the growing impact of cloud-native adoption and AI/ML workloads on the environment, stressing the need for carbon-aware computing. This concept extends beyond mere cost efficiency, sometimes requiring organizations to pay slightly more for less carbon. It encompasses three pillars: carbon intensity, energy efficiency, and measuring impact.

• Carbon Intensity: Choosing cloud providers and regions based on their energy mix. Examples include Leaf Cloud in the Netherlands and Civo in the UK, which use dislocated data centers to reheat buildings or public spaces. Bianchi specifically noted Sweden as a European region with the "best and cleanest energy" due to its resource mix.
• Energy Efficiency: Optimizing code and workflows to reduce CPU cycles, recognizing that even small software optimizations can have a greater environmental impact than individual recycling efforts. Practices like multi-tenancy and efficient containerization (e.g., slimming down containers) contribute to greener operations.
• Measuring Impact: Utilizing tools and initiatives to quantify the carbon footprint of workloads. Cube Green was introduced as a project that enables automatically turning off test environments during off-peak hours (e.g., weekends, overnight), saving both money and carbon emissions. The CNCF TAG Sustainability initiative for carbon reviews aims to measure the carbon footprint of CNCF projects, fostering green open source practices.

Bianchi emphasized that while economic incentives for sustainable coding practices are less pronounced in open source, community engagement through opening issues and advocating for carbon efficiency can drive change.

GitOps

Steve Wade, a vocal proponent of GitOps, described it as a fundamental shift from Infrastructure as Code (IaC). While IaC focuses on version controlling infrastructure definitions, GitOps elevates Git to the single source of truth for both application workloads and their configurations. This paradigm enforces an operational mindset where manual intervention is minimized in favor of automated deployments triggered by Git commits.

The sweet spots for GitOps are numerous:

• Kubernetes Management: Centralizing thousands of YAML files in Git repositories (either monorepos or split by function, e.g., security features in a security repo, product releases in team repos).
• Multi-cluster Consistency: Simplifying the deployment and management of multiple clusters from a singular repository with environment overlays or multiple specialized repositories. This allows for easier detection of drift between environments (e.g., staging vs. production) and ensures consistency.
• Auditability and Compliance: For regulated industries like financial institutes, GitOps provides an invaluable audit trail. Every change is a Git commit, offering a clear review process with approvers and a comprehensive history that can be used to generate compliance reports.

The principle is that any change to the system must be initiated via a Git pull request, reviewed, approved, and merged, providing a robust, auditable, and consistent deployment pipeline.

Demo / Proof of Concept

▶ Watch: Spinning wheel reveals Sustainability and Service Mesh topics (7:00)

The "Cloud Native Wheel of Fortune" session itself served as a unique demonstration of interactive audience engagement rather than a traditional technical proof of concept for a specific tool or technology. The "demo" involved the live spinning of a digital wheel, populated with pre-selected and audience-suggested cloud-native topics, followed by real-time Q&A via Slido. This allowed the speakers to dynamically adapt their content to the most pressing interests of the KubeCon EU attendees. While no code was run or specific software was showcased in a live technical demonstration, the format effectively proved that an interactive approach can foster deeper engagement and directly address audience needs, making the talk highly relevant and personalized.

Defensive Implications

▶ Watch: Matteo introduces EVPF for kernel-level observability (8:00)

The insights from the "Cloud Native Wheel of Fortune" offer several critical defensive implications for organizations operating in the cloud-native landscape:

• Robust Image Security for Air-gapped Environments: For air-gapped or highly regulated setups, defenders must prioritize the secure deployment and management of container image registries. Implementing strict provenance checks and leveraging SBOMs (Software Bill of Materials) are crucial to ensure that all images, even those used for bootstrapping Kubernetes, are free from known vulnerabilities and supply chain risks. Network segmentation and careful resource planning are paramount.
• Leverage eBPF for Deep Security Observability: Defenders should explore and adopt eBPF-based tools like Falco to gain unparalleled kernel-level visibility into runtime activities. This enables the detection of suspicious behaviors, unauthorized process execution, file system changes, and network anomalies that traditional security tools might miss. Integrating eBPF security with existing alerting systems (ee.g., Slack, Teams) provides real-time incident response capabilities, as demonstrated by the 25% reduction in security incidents in the fashion company example.
• Strategic Scalability for Security and Stability: When scaling Kubernetes, security must be designed in from the ground up. Implementing strong multi-tenancy with robust namespace isolation, resource quotas, and meticulously crafted network policies is essential. For complex, distributed environments, a service mesh can provide a consistent and scalable mechanism for enforcing security policies across multiple clusters, reducing the burden on individual application teams and improving overall security posture. Dedicated clusters for specific functions can also minimize blast radius.
• Embrace Carbon-Aware Security and Operations: The environmental impact of cloud infrastructure is a growing concern. Defenders should advocate for and implement carbon-aware computing practices. This includes optimizing resource usage, leveraging tools like Cube Green to power down non-production environments during off-peak hours, and prioritizing cloud regions with cleaner energy mixes. While not directly a security control, reducing unnecessary resource consumption strengthens overall operational resilience and aligns with broader corporate responsibility goals.
• GitOps as a Foundation for Security and Compliance: GitOps provides a powerful defensive framework by making Git the single source of truth for all infrastructure and application configurations. This ensures every change is version-controlled, auditable, and subject to a formal review and approval process. This inherent auditability is critical for compliance requirements (e.g., in financial institutions) and provides a clear history for forensic analysis during security incidents. It drastically reduces the risk of configuration drift and unauthorized manual changes, leading to a more secure and stable environment.

In essence, the talk reinforced that effective cloud-native defense requires a holistic, interconnected approach that integrates security considerations into every aspect of design and operation, from image provenance and kernel-level monitoring to scalable policy enforcement and auditable configuration management.

Key Takeaways

• Cloud Native is Interconnected: No single technology operates in isolation. Successful cloud-native adoption requires understanding how diverse tools and concepts (like eBPF, GitOps, and scalability) interrelate and impact each other.
• Air-gapped Environments Demand Rigorous Planning: Securely managing container images with SBOMs, establishing robust networking, and addressing resource constraints are paramount challenges in air-gapped Kubernetes deployments.
• eBPF Revolutionizes Observability and Security: eBPF provides unprecedented kernel-level visibility and control without modifying the kernel, enabling powerful tools like Falco to significantly enhance runtime security and detect anomalies.
• Scalability Requires Holistic Strategy: Effective Kubernetes scaling involves optimizing control planes, simplifying developer experience, managing geographical distribution, and implementing smart multi-tenancy with tools like vcluster and service meshes.
• Sustainability is a Core Cloud Native Responsibility: Adopting carbon-aware computing, optimizing resource usage with tools like Cube Green, and selecting green cloud regions are crucial steps toward reducing environmental impact.
• GitOps Ensures Auditability and Consistency: Leveraging Git as the single source of truth for all configurations simplifies multi-cluster management, enforces audit trails, and strengthens compliance in cloud-native operations.

About the Speaker(s)

Steve Wade is introduced as a Cloud Native Catalyst from the UK. His professional background is deeply rooted in platform engineering, encompassing extensive experience in running large-scale Kubernetes clusters, debugging complex service meshes, and navigating various other aspects of cloud-native infrastructure. He highlighted his work in highly regulated environments, such as financial institutions, which informed his insights on challenges like air-gapped deployments and compliance.

Matteo Bianchi is a CNCF Ambassador and works as a Solution Engineer at GitHub. He humorously describes himself as a "lazy engineer," which he clarifies means he "likes to automate stuff," particularly in areas like CI/CD. Bianchi is known for his ability to simplify complex topics, as evidenced by his "eBPF to my grandma" series of talks, making advanced concepts like eBPF accessible to a broader audience. He brings a perspective on security and efficiency, drawing from his consulting experience with large organizations.