A Day in the Life of a Kubernetes Engineer

A Day in the Life of a Kubernetes Engineer

Speakers: Shane, Staff Platform Engineer, Shopify; Nikita, Principal Engineer, Broadcom; Casper, Developer Relations, Lunar; Micah Hler, Principal Engineer, AWS; Rajas, Broadcom

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=QhTlZs4m59w

Overview

This panel discussion at KubeCon EU, titled "A Day in the Life of a Kubernetes Engineer," offered a candid and insightful look into the often-unseen challenges faced by the individuals responsible for maintaining and evolving Kubernetes clusters. While the industry frequently celebrates the successes and transformative power of Kubernetes, this session deliberately shifted focus to the human element – the engineers who navigate its complexities, confront its failures, and continuously push its boundaries. The panel, composed of seasoned Kubernetes contributors, maintainers, and early adopters, shared personal anecdotes, technical insights, and philosophical perspectives on what it truly means to be a Kubernetes engineer.

The discussion highlighted that the role is far from static, evolving significantly since Kubernetes' nascent stages. It's a role characterized by constant learning, problem-solving, and a nuanced understanding of both technology and business objectives. The panelists emphasized that while Kubernetes offers immense power, it also introduces unique challenges, ranging from managing YAML files to navigating backward compatibility breaks and mitigating security vulnerabilities. This talk is crucial for anyone involved in the Kubernetes ecosystem, offering a grounding perspective that balances the hype with the hard realities of operational excellence and continuous improvement.

Background

▶ Watch: Panel introduction: The real issues of a Kubernetes engineer (0:00)

The panelists' journeys into Kubernetes underscore the technology's rapid evolution and the diverse paths individuals took to become experts in a then-emerging field. Shane, for instance, transitioned from a risk-averse managed security services role to Shopify in 2017, drawn by the pioneering work on Kubernetes alongside Google teams. His experience reflects the early excitement of "building those things with them" when core concepts like controllers and Custom Resource Definitions (CRDs) were either nascent or non-existent, often relying on "third-party resources" instead.

Nikita, a principal engineer at Broadcom and a long-time Kubernetes maintainer, began through a Google Summer of Code internship, contributing directly to features like CRDs. This path highlights the academic and open-source contribution route to expertise. Casper, now in developer relations at Lunar, entered the scene even earlier in 2015 while writing his master's thesis. His ambitious project involved running microservices on a Raspberry Pi Kubernetes cluster, demonstrating an early, hands-on exploration of the technology's potential for distributed systems.

Micah Hler, a principal engineer at AWS and a Kubernetes security committee member, ran his first cluster in 2016. This was a period of "container orchestration wars," where alternatives like Mesos and Docker Swarm competed for dominance. Micah's experience involved "hand rolling it out" on AWS, predating tools like kubeadm and kops, and recognizing Kubernetes as a solution to his orchestration problems at a small startup. Rajas, the moderator, also brings a strong background as a Kubernetes contributor active in various CNCF technical advisory groups.

Collectively, their stories paint a vivid picture of Kubernetes' foundational years: a time of intense innovation, manual configuration, significant learning curves, and a community actively shaping the future of cloud-native infrastructure. The transition from rudimentary bash scripts and manual operations to sophisticated GitOps practices and platform engineering solutions forms a core part of this historical context, illustrating the maturity gained over nearly a decade.

Key Findings

▶ Watch: Nikita's start: Google Summer of Code to Kubernetes maintainer (2:29)

The panel discussion yielded several key insights into the identity, challenges, and evolution of a Kubernetes engineer:

1. Defining a Kubernetes Engineer is a Spectrum: There's no single definition. It encompasses anyone "driving Kubernetes," from upstream contributors and maintainers to end-users and advocates. The role's nature depends heavily on career stage, company size, and specific problems being solved. As engineers mature, their focus often shifts from merely editing YAML files to designing platforms, automating processes, and ultimately questioning if Kubernetes is even the right tool for a given business problem. This emphasizes a move from tactical execution to strategic problem-solving.
2. Business Acumen is Crucial: A recurring theme was the importance for Kubernetes engineers to understand the business context and goals. Technology, including Kubernetes, is a tool to solve business problems, not an end in itself. Engineers, especially as they advance, must consider how their work provides value to customers and aligns with organizational objectives, enabling them to communicate effectively with executives who seek "business solutions, not engineering solutions."
3. Misconceptions Abound:

• "Kubernetes Engineers love YAML": This was quickly debunked, with many panelists and audience members expressing a desire to abstract away YAML complexity for developers, indicating a push towards better user experience and higher-level abstractions in the next decade of Kubernetes adoption.
• "Kubernetes is a Panacea": The belief that Kubernetes is the hammer for every nail is a dangerous misconception. While powerful for specific use cases (e.g., stateless web applications requiring rapid scaling and frequent changes), it's often the wrong tool for others (e.g., a simple SQL database on a single VM). Pragmatism and understanding Kubernetes' strengths and weaknesses are vital.

1. Human Error and Production Failures are Inevitable Learning Opportunities: The panelists openly shared stories of significant production outages caused by human error, such as Shane accidentally draining an entire cluster in parallel or Casper's nginx-ingress-controller deployment causing traffic loss. These anecdotes highlighted the importance of acknowledging mistakes, learning from them, and implementing robust processes and automation (like GitOps) to minimize future occurrences.
2. Community Involvement and Staying Updated are Essential: Given Kubernetes' rapid development, keeping track of upstream changes is a significant challenge. The panel recommended active participation in Special Interest Groups (SIGs), reading SIG notes, reviewing release notes (including in-progress branches), and following the official Kubernetes blog for high-level summaries. Local meetup groups and peer discussions were also cited as valuable for contextualizing changes and sharing experiences.

Technical Deep Dive

▶ Watch: Casper's unique path: Master's thesis on Raspberry Pi Kubernetes (3:26)

The panel discussion, while broad in scope, offered several technical insights gleaned from years of hands-on experience with Kubernetes, particularly highlighting the evolution of practices and the challenges faced in its early days.

Early Kubernetes (2015-2017) was characterized by a lack of mature tooling. Micah Hler described "hand rolling it out" on AWS, prior to the existence of kubeadm or kops. Core concepts like Custom Resource Definitions (CRDs) didn't exist, with "third-party resources" serving a similar, less stable purpose. This era demanded deep technical understanding and often bespoke solutions.

Shane's anecdote from Shopify underscored the difficulties of cluster upgrades in a less mature environment. The absence of sophisticated tooling meant manual, script-based operations for thousands of nodes. His critical error involved using a parallel script for drain operations instead of a sequential one, leading to the rapid disappearance of an entire cluster. This incident highlighted:

• Backward Compatibility Issues: Kubernetes often introduces breaking changes, even between minor versions, making upgrades a precarious task.
• The Perils of Manual Operations: Reliance on bash scripts and individual engineers running commands from their laptops introduced significant risk.

Casper's experience with an nginx-ingress-controller upgrade causing traffic loss further illustrated the fragility of early deployments when using raw kubectl apply commands. These incidents collectively pushed the community towards more robust deployment strategies.

Nikita's story as a maintainer provided a deeper look into debugging and fixing issues at the core Kubernetes level. A job managing an authentication service was failing due to pods crashing immediately. After extensive log analysis and configuration checks, the root cause was traced to a change in a feature gate within the Kubernetes codebase, which altered the backoff strategy from exponential to linear. This meant the service was not resilient enough to infrastructure issues. The solution required digging into the job controller code and contributing a fix directly to upstream Kubernetes, emphasizing the need for deep code understanding when standard debugging fails.

Micah's security contribution revealed a port forwarding vulnerability in Kubernetes. An attacker could overwrite a node's IP address with a cloud credential IP, allowing port forwarding to the API server's cloud credentials. His initial fix, while addressing the immediate CVE, inadvertently introduced a time-of-check time-of-use (TOCTOU) bug related to DNS resolution, which was later discovered and patched. This demonstrates the intricate security challenges and the continuous vigilance required even by experienced contributors.

The solutions and advancements discussed reflect a significant maturation of Kubernetes practices:

• GitOps: This paradigm, where desired state is declared in Git and continuously synchronized to clusters, emerged as a critical safeguard against human error from manual kubectl apply commands.
• Platform Engineering: The concept of building a platform on top of Kubernetes to abstract its complexity from developers was repeatedly emphasized. This includes automating YAML generation, validation, and deployment processes to improve user experience and embed best practices.
• Automated Cluster Upgrades: Shane described Shopify's current sophisticated system, involving PR-triggered automation, staging and Canary clusters, and soak times (periods to observe stability) before rolling out changes to production. This multi-layered approach provides redundancy for redundancy, minimizing disruption.
• Strategic Tool Selection: The panel stressed that Kubernetes is a specialized tool. It excels at deploying stateless web applications that require rapid scaling and frequent changes but is often overkill or even detrimental for simpler, stateful applications like a single SQL database on a VM.

These technical discussions highlight a shift from merely operating Kubernetes to engineering comprehensive solutions that leverage its power while mitigating its inherent complexities and risks through automation, abstraction, and a deep understanding of its internals.

Demo / Proof of Concept

▶ Watch: Defining a Kubernetes Engineer: The core discussion begins (6:05)

This panel discussion did not include a live demonstration or a proof of concept. The format was an open conversation where panelists shared their experiences, insights, and lessons learned from their long careers working with Kubernetes.

Defensive Implications

▶ Watch: Panelists define a Kubernetes Engineer: Frustration and car analogy (6:33)

The collective wisdom shared by the panelists offers crucial defensive implications for organizations operating Kubernetes at scale:

1. Embrace GitOps for All Infrastructure Changes: The anecdotes of production failures due to manual kubectl commands strongly advocate for a strict GitOps methodology. All infrastructure and application configurations should be version-controlled, reviewed via pull requests, and automatically applied to clusters. This minimizes human error, provides an audit trail, and enables easy rollbacks.
2. Implement Robust Multi-Stage Deployment and Upgrade Strategies: Learn from Shopify's advanced approach:

• PR-triggered automation: Automate deployments and upgrades based on Git commits.
• Staging and Canary Clusters: Deploy changes to non-production environments first, then to small, isolated production subsets (Canary) before a full rollout.
• Soak Times and Automated Checks: Introduce mandatory observation periods and automated health checks at each stage to detect issues before they impact a wide audience. This provides "redundancy for redundancy."

1. Invest in Platform Engineering to Abstract Complexity: To improve developer experience and reduce the likelihood of configuration errors (like incorrect YAML indentation), build a platform on top of Kubernetes. This platform can automate YAML generation, validate configurations, and embed best practices, allowing developers to focus on business logic rather than Kubernetes primitives.
2. Foster Deep Understanding of Kubernetes Internals: As demonstrated by Nikita's debugging of a job controller bug and Micah's security contributions, sometimes the only way to solve critical problems is to delve into the Kubernetes codebase. Encourage engineers to understand core controllers, feature gates, and the API server's behavior. This deep knowledge is invaluable for advanced debugging, performance tuning, and contributing upstream fixes.
3. Prioritize Resilience and Redundancy: Design systems to withstand cluster-wide failures. While Shane's cluster disappearance was catastrophic, the eventual solution involved "fleets of clusters" and the ability to shift traffic, highlighting the importance of architectural resilience beyond single-cluster boundaries.
4. Stay Actively Engaged with the Upstream Community: Kubernetes evolves rapidly. Defenders must stay informed about changes, deprecations, and new features. This involves:

• Participating in Special Interest Groups (SIGs).
• Reading SIG notes and release notes (for current and upcoming versions).
• Following the official Kubernetes blog for high-level summaries.
• Leveraging local meetups and peer discussions to contextualize changes.

1. Be Vigilant About Security Vulnerabilities: Micah's experience with a port forwarding vulnerability and a subsequent TOCTOU bug underscores the continuous need for security awareness. Regularly review security advisories, patch clusters promptly, and understand common attack vectors in cloud-native environments.
2. Practice Pragmatic Tool Selection: Avoid the misconception that Kubernetes is a universal solution. Understand its strengths (e.g., scaling stateless web applications, microservices orchestration) and weaknesses (e.g., complexity for simple workloads, stateful application challenges). Use Kubernetes where it provides clear value and consider simpler alternatives where appropriate.

Key Takeaways

• Kubernetes Engineering is Evolving: The role of a Kubernetes engineer is dynamic, shifting from tactical YAML management to strategic platform design and business problem-solving.
• Business Context is Paramount: Engineers must understand how Kubernetes serves business goals, not just technical ones, to ensure long-term value and relevance.
• Pragmatism Over Dogma: Kubernetes is a powerful tool, but it's not a panacea; selecting the right tool for the specific problem is crucial to avoid unnecessary complexity.
• Automate Everything, Especially Upgrades: GitOps, multi-stage deployments with Canary releases, soak times, and automated checks are essential to mitigate human error and ensure resilience during cluster operations.
• Community Engagement is Key for Staying Current: Active participation in SIGs, reading release notes, and leveraging community discussions are vital for navigating Kubernetes' rapid development.
• Embrace Failures as Learning Opportunities: Production incidents, though challenging, provide invaluable lessons that drive the adoption of more robust processes, automation, and platform improvements.

About the Speaker(s)

• Shane: Started his role as a Kubernetes engineer at Shopify in 2017 after working for a managed security services department in a consulting company. He joined Shopify to work on "new fangled Kubernetes thing" and collaborate with teams from Google.
• Nikita: A Principal Engineer at Broadcom and a Kubernetes maintainer for a while. He got started by building a lot of features for CRDs in Kubernetes itself. He has been a part of the CNCF technical oversight committee and a co-chair for KubeCon in the past. His journey began with a Google Summer of Code internship.
• Casper: Currently works in developer relations, but used to be a staff platform engineer at Lunar. He got into Kubernetes in 2015 while writing his master's thesis, exploring running microservices on a Raspberry Pi Kubernetes cluster.
• Micah Hler: A Principal Engineer at AWS and a Kubernetes contributor. He is on the Kubernetes security committee and is a co-chair for SIG O. He started with Kubernetes as an engineer way back in 2016, running his first cluster on AWS before tools like cops and kubeadm existed, and later started contributing to the project while working at a startup.
• Rajas: Works at Broadcom doing all things Kubernetes. He is also a contributor of Kubernetes and is currently active in CNCF technical advisory groups for runtime, working group artificial intelligence, and other initiatives.