Tutorial: "Working Code Wins": Win Big With a Cloud Native Hackathon Start... P. Morton & A. Bangser

"Working Code Wins": Win Big With a Cloud Native Hackathon Start... P. Morton & A. Bangser

Speakers: P. Morton; A. Bangser

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=jid0uSnNku8

Overview

This KubeCon EU tutorial, "Working Code Wins: Win Big With a Cloud Native Hackathon Start," presented by P. Morton and A. Bangser, delves into the transformative power of platform engineering in accelerating developer experience (DevX), particularly within the context of internal hackathons. The talk showcases how The Access Group, one of the UK's largest software companies, leveraged Kratics, an open-source platform orchestration framework, alongside other cloud-native tools to streamline the provisioning of development environments and applications. Morton, an engineer at The Access Group, shares his journey of building a robust, self-service platform that significantly reduced the operational overhead traditionally associated with hackathons, while Bangser, a CNCF ambassador and platform engineer, provides the foundational theoretical context for this shift.

The core message of the presentation revolves around the concept of platform engineering as a strategic re-centralization of operations, designed to provide developers with "as a Service" capabilities without sacrificing the autonomy and speed gained from the DevOps movement. By establishing codified "Promises" that define and automate the delivery of platform services—from databases to entire applications—the speakers illustrate how organizations can create a sustainable, scalable, and secure internal developer platform. This approach not only empowers developers by giving them self-service access to infrastructure but also ensures consistency, enforces guardrails, and simplifies maintenance, ultimately fostering a culture where "working code wins" by enabling rapid iteration and innovation.

Background

▶ Watch: Introduction to talk and speakers' engineering philosophies (0:00)

The genesis of this talk lies in the inherent challenges of traditional hackathons and the broader evolution of software development practices. Hackathons, while excellent for fostering innovation and learning, often entail significant stress and manual effort for organizers to provision and manage development environments. This frequently involves bespoke setups, inconsistent configurations, and considerable "eating stress for breakfast" for the platform team, as Morton vividly describes his experience. The goal was to transform this chaotic process into a seamless, self-service experience, reducing friction and maximizing the time developers spend on coding.

From a broader industry perspective, the talk frames its solution within the ongoing evolution from DevOps to Platform Engineering. DevOps successfully decentralized operations, promoting autonomy and reducing delays by embedding operational responsibilities within development teams. However, this often led to fragmented practices, inconsistent security postures, and a loss of economies of scale. Platform Engineering seeks to address these drawbacks by intentionally recentralizing certain operational aspects, not by reverting to bureaucratic bottlenecks, but by providing centralized management with decentralized consumption. The objective is to build "as a Service" platforms that offer developers a curated, self-service experience with built-in guardrails, security, and support, thereby improving efficiency and maintainability without compromising developer autonomy. This paradigm shift necessitates a robust underlying framework capable of defining, orchestrating, and delivering these services declaratively and consistently.

Key Findings

▶ Watch: Kicking off the hands-on lab and accessing the workshop platform (2:40)

The central discovery and contribution of this talk lie in demonstrating a practical, cloud-native approach to building a self-service internal developer platform, specifically tailored for high-velocity events like hackathons. The speakers highlight several key findings:

1. Kratics as an Orchestration Hub: Kratics emerges as the pivotal technology, functioning as an orchestration layer that simplifies complex cloud-native deployments. It enables platform teams to define and manage services as reusable, composable units, abstracting away underlying infrastructure complexity from application developers.
2. The Power of "Promises": The concept of a "Promise" is introduced as a fundamental building block. A Promise in Kratics is a Custom Resource Definition (CRD) in Kubernetes that defines an API contract for a platform service. This contract specifies what users can request, what parameters are allowed, and what constraints apply. This allows platform engineers to codify policies and best practices, offering a simplified, opinionated interface to complex underlying services (e.g., a Postgres database).
3. GitOps for Declarative and Recoverable State: The adoption of GitOps is critical. Kratics integrates with GitOps workflows by writing declarative configuration (YAML files) to a Git repository (gitt in the demo), which is then picked up and applied to the infrastructure by tools like Flux. This ensures that the platform's state is version-controlled, auditable, and inherently recoverable, as demonstrated by Morton's experience of recovering from a deleted cluster by re-applying the Git state.
4. Composition for Scalability and Flexibility: The platform architecture emphasizes composition, allowing smaller, specialized promises (like a Postgres promise) to be combined into higher-level, more powerful services (like an App-as-a-Service promise that orchestrates both a database and a web application). This modularity supports both "fruit salad" (individual components) and "fruit basket" (pre-composed solutions) approaches, providing flexibility to meet diverse developer needs.
5. Enhanced Developer Experience with Backstage: Integrating a developer portal like Backstage with Kratics significantly improves the DevX. Backstage provides a user-friendly frontend for interacting with the platform's APIs. Kratics can translate its internal promise definitions into Backstage-shaped YAML, making services discoverable and consumable through a familiar UI, CLI, or even automated ticketing systems, meeting developers "where they are."
6. Built-in Reconciliation and Disaster Recovery: By leveraging Kubernetes' inherent reconciliation loop and GitOps principles, the platform gains significant resilience. Workflows triggered by Promises are run as Kubernetes jobs, and the declarative state in Git ensures that if a cluster is destroyed, it can be rebuilt to its last known good state, including the re-deployment of all managed applications and services.

Technical Deep Dive

▶ Watch: Guide to navigating and using the Instruct virtual lab (3:40)

The technical foundation of this platform relies heavily on Kratics operating within a Kubernetes ecosystem, augmented by GitOps principles and a Backstage frontend.

At its core, Kratics extends Kubernetes capabilities by introducing Promises. A Promise is essentially a Custom Resource Definition (CRD) that defines a user-facing API for a platform service. When a user requests a service via this API (e.g., a Postgres database), Kratics orchestrates the underlying infrastructure. The Promise specification includes:

• API Field: This defines the contract with the user, specifying parameters like name, environment, namespace, and team ID. This allows platform engineers to create a simplified, opinionated API, shielding application developers from the complexity of numerous flags and configurations of an underlying operator. For instance, the Postgres Promise showcased a very simplified API compared to the full configurability of a Postgres operator.
• Workflows Field: This defines the business logic executed when a user requests a service. Workflows are run as Kubernetes Jobs and can consist of multiple containers, allowing logic to be written in any language (bash, Golang, Python, Elixir). These workflows perform tasks such as security checks, authorization, cost management, manual sign-offs, and crucially, the deployment of resources.
• Destination Selectors Field: This dictates where the declarative code generated by the workflow should be written. It uses Kubernetes-like labels and selectors to target specific Git repositories (e.g., environment: dev) and clusters (e.g., a worker cluster versus the platform cluster where Kratics runs).

The deployment mechanism heavily relies on GitOps. When a workflow executes, it generates declarative YAML files for the required resources (e.g., Kubernetes Deployments, Services, ConfigMaps, or even custom resources for operators). These YAML files are then committed to a designated Git repository (referred to as gitt in the demo, mimicking a Git server). A GitOps agent like Flux (mentioned explicitly) monitors this repository and applies the changes to the target Kubernetes cluster. This ensures that all infrastructure is managed declaratively, is version-controlled, and changes are auditable.

For example, when a user requests a Postgres database through the Postgres Promise:

1. Kratics receives the request via its simplified API.
2. A workflow is triggered, running as a Kubernetes Job.
3. This workflow generates the necessary YAML for a Postgres operator to deploy and manage a Postgres instance.
4. The generated YAML is committed to the specified Git repository by the destination selector.
5. Flux detects the change in Git and applies the YAML to the target worker cluster, causing the Postgres operator to provision the database.

The concept of composition is vital for building complex services. The talk demonstrates an App-as-a-Service promise that orchestrates multiple underlying promises. This higher-level promise depends on and makes requests to other promises, such as the Postgres promise and an Nginx promise (lines 62-66 in the demo code). This allows platform engineers to define an opinionated application stack (e.g., a web app with a database) as a single, consumable unit, while still leveraging the standardized, lower-level services.

Environment variable configuration is handled dynamically. When a user specifies environment variables for their application, Kratics workflows generate a ConfigMap containing these variables. This ConfigMap is then attached to the application's container, providing a standardized and automated way to inject configuration. The ability to "mutate" files by placing workflow steps at specific points (e.g., "sandwich in the middle or at the end") allows for flexible configuration generation and modification.

Finally, the integration with Backstage elevates the developer experience. Kratics can generate Backstage-shaped YAML from its promise definitions, which is then stored in a state store like MinIO (an S3-compatible object storage). Backstage reads this YAML, populating its catalog with the available services and scaffolding templates. This allows application developers to discover, request, and provision applications directly from the Backstage portal, which then makes an API call to Kratics, ensuring all requests flow through the consistent platform API. The hackathon instance itself ran on a Backstage instance provisioned via the App-as-a-Service promise, using a Postgres database configured through the platform, demonstrating "eating your own dog food."

Demo / Proof of Concept

▶ Watch: Understanding paced learning and available workshop support (5:30)

The entire talk was structured as a hands-on workshop using Instruct, a virtual lab environment, demonstrating the capabilities of the platform in real-time.

1. Postgres-as-a-Service:

• Participants started by installing a Kratics Promise for a Postgres database.
• They then made a request for a database resource, specifying simple parameters like name, environment (e.g., dev), namespace, and team ID.
• The workflow pipeline was shown to execute, generating declarative YAML that instructed a Postgres operator to spin up the database on a designated worker cluster.
• The speakers highlighted how this abstracts away the complexity of the operator, allowing platform engineers to define strict policies (e.g., prod environments automatically get backups and replicas) that are enforced by changing a single tag. Morton noted that he "fell in love" at this point, realizing the ease of provisioning "100 databases" with minimal input.

1. App-as-a-Service (To-Do App):

• This section built upon the previous one by introducing a higher-level promise: App-as-a-Service. This promise composed the Postgres promise with an Nginx promise to deliver a complete application stack.
• Users requested a To-Do app, providing inputs like the application name, container image, whether a database was required (and its type), and the service port.
• The workflow orchestrated the deployment of both the web application and its backing Postgres database, demonstrating how developers can provision a full application without individually managing each component.
• The result was a functional To-Do app, accessible via a URL exposed in the Instruct environment, showcasing the end-to-end self-service experience.

1. Environment Variable Configuration:

• The demo then showed how to extend the App-as-a-Service promise to accept environment variables.
• An updated promise specification included an environment API field.
• When applied, a workflow script (scripts/environment_ve_configure) generated a ConfigMap from user-provided environment variables, which was then attached to the application's container. This illustrated the power of mutating existing configurations within the workflow pipeline.

1. Backstage Integration:

• The workshop introduced Backstage as the developer portal frontend.
• Kratics was configured to translate its promise definitions into Backstage-shaped YAML, storing it in MinIO.
• This allowed Backstage to automatically populate its catalog with the App promise and the deployed To-Do app.
• Participants used Backstage's scaffolding template to create a new application (named "kubecon" in the demo), demonstrating how Backstage makes API calls to Kratics, providing a user-friendly interface for provisioning. This highlighted how requests can originate from a GUI, CLI, GitOps, or even a ticketing system, all funneling through the consistent platform API.

1. Hackathon Success Story:

• Morton shared real-world results: the platform ran their internal Backstage instance for the hackathon, supporting over 250 projects across three different hacks (product, engineering, professional services).
• They customized the App-as-a-Service promise to include Let's Encrypt SSL certificates for all applications, providing a fully secure and polished experience.
• A critical learning moment involved Morton accidentally deleting all resources the day before a hackathon. Thanks to GitOps and a diligent teammate's advice to back up the database, he was able to recover the entire platform by re-applying the declarative state from Git, showcasing the inherent disaster recovery capabilities.
• He also mentioned using QSSH, an open-source tool, to automatically update container images via webhooks, simplifying the developer workflow during the hackathon by removing the need for teams to understand complex GitOps flows for simple image updates.

Defensive Implications

▶ Watch: Live demo troubleshooting and fixing a workshop platform issue (6:40)

The platform engineering approach demonstrated in this talk presents significant defensive implications, primarily centered around centralized control, automated guardrails, and enhanced resilience.

1. Codified Security and Policy Enforcement: By defining services through Kratics Promises, platform engineers can embed security policies and best practices directly into the platform's API. This means that when an application developer requests a database, the underlying promise can automatically ensure it's provisioned with appropriate security configurations, network segmentation, and compliance requirements. For instance, a "prod" environment tag can automatically trigger configurations for backups, multiple replicas, and specific database operators that meet enterprise standards, preventing developers from inadvertently deploying insecure or non-compliant infrastructure. This shifts security left, building it into the self-service mechanism rather than relying on manual checks post-deployment.

1. Reduced Attack Surface through Abstraction: The simplified API exposed by Promises abstracts away the complex underlying infrastructure. Application developers interact with a high-level contract, not directly with Kubernetes manifests or operator configurations. This reduces the number of knobs developers can turn, minimizing misconfigurations that could introduce vulnerabilities. It also limits direct access to sensitive infrastructure, channeling all provisioning through a controlled, auditable API gateway.

1. Consistent and Auditable Deployments via GitOps: The reliance on GitOps for all deployments means that the desired state of the entire platform, including all provisioned applications and services, is declaratively stored in a version-controlled Git repository. This provides an immutable audit trail of all changes, making it easier to track who changed what, when, and why. In the event of a security incident, this historical record is invaluable for forensic analysis and understanding the blast radius.

1. Enhanced Disaster Recovery and Resiliency: The combination of GitOps and Kubernetes' reconciliation loop provides a robust disaster recovery strategy. As Morton vividly illustrated with his pre-hackathon cluster deletion incident, if an entire cluster or data center is lost, the platform can be rebuilt by simply reapplying the declarative state from Git. Kratics workflows, running as Kubernetes Jobs, ensure that services are continuously reconciled to their desired state. This inherent self-healing capability minimizes downtime and recovery time objectives (RTO), making the platform more resilient to catastrophic failures or accidental deletions.

1. Fleet Management for Consistent Updates: The platform enables fleet management, allowing platform teams to introduce continuous changes and updates to their promises. When a promise is updated (e.g., adding a new environment variable or updating a security patch for an underlying operator), Kratics can automatically replay and update all previous resource requests. This ensures that all applications and services managed by the platform remain up-to-date with the latest best practices and security patches, significantly reducing the burden of manual updates and maintaining a consistent security posture across the entire fleet of applications.

1. Centralized Observability and Monitoring: While not explicitly detailed in the transcript, a centralized platform naturally facilitates centralized logging, monitoring, and observability. By standardizing how services are provisioned and run, platform teams can integrate consistent tooling to gain comprehensive insights into the health, performance, and security of all applications, making it easier to detect and respond to anomalies.

Key Takeaways

• Platform Engineering empowers developers with self-service capabilities while providing centralized control and guardrails. This strategic re-centralization aims to achieve economies of scale and improve security without sacrificing the autonomy and speed gained from DevOps.
• Kratics "Promises" define the contract between platform engineers and application developers. These are Kubernetes CRDs that offer simplified, opinionated APIs for complex services, enabling platform teams to codify policies and best practices.
• GitOps is fundamental for declarative, auditable, and resilient platform operations. By storing the desired state in Git and using tools like Flux, deployments are consistent, version-controlled, and the platform can be recovered from catastrophic failures.
• Composition allows for building powerful "as a Service" offerings from smaller, reusable components. This modularity enables platform engineers to create comprehensive application stacks (e.g., App-as-a-Service) while maintaining flexibility for individual service consumption.
• Developer portals like Backstage significantly enhance the developer experience by providing a user-friendly interface to platform APIs. Integrating Backstage with Kratics makes platform services discoverable and consumable through a familiar UI, CLI, or even automated ticketing systems.
• The combination of Kubernetes' reconciliation loop and GitOps inherently provides robust disaster recovery capabilities. The platform can automatically detect and correct deviations from the desired state, and a complete rebuild from Git is possible if a cluster is lost.

About the Speaker(s)

P. Morton is an engineer at The Access Group, a major UK-headquartered software company. He described himself as a "GUI person" who loves automation and actively seeks "a better way" in software engineering. His personal journey, learning Kubernetes, Kratics, and GitOps on "hard mode," underpinned the practical, hands-on nature of the hackathon platform he built. His enthusiasm for the subject is evident, sharing personal anecdotes like enjoying rum while watching pipelines kick off during hackathons and recovering from accidentally deleting a cluster through GitOps.

A. Bangser is a prominent figure in the cloud-native community, serving as a CNCF ambassador and Team Topologies advocate. Her career has been dedicated to internal tooling, focusing on improving developer experience (DevX) for her colleagues across various roles including QA, DevOps, SRE, and Platform Engineer. Her mantra, "Stand for what you stand for, not against," reflects her positive approach to technology adoption and community engagement. She provided the theoretical framework and architectural insights, contextualizing Morton's practical implementation within broader platform engineering principles.