Tutorial: Workshop: Developing as a Team for Kubernetes With Nix an... Leigh Capili & Tanja Ulianova

Tutorial: Workshop: Developing as a Team for Kubernetes With Nix and GitOps

Speakers: Leigh Capili, Staff Software Engineer, Control Plane; Tanja Ulianova

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=NnYtnUeJi7U

Overview

This talk, presented as a hands-on tutorial by Leigh Capili and Tanja Ulianova at KubeCon EU, delves into the powerful synergy of Nix and GitOps principles to revolutionize the development and deployment of applications for Kubernetes. The core premise is to extend the benefits of declarative, reproducible infrastructure management, typically associated with GitOps in production, all the way left to the developer's machine and local environment. Capili, a maintainer of the Flux project, demonstrates how to achieve truly hermetic and consistent development environments using Flux (the usability layer on top of Nix, developed by Flocks) to manage dependencies, build applications, and containerize them for Kubernetes deployments.

The presentation addresses a fundamental challenge in software development: ensuring environmental consistency across developer machines, CI/CD pipelines, and production. By integrating Nix's functional package management capabilities with GitOps workflows, the talk outlines a path to eliminate common "works on my machine" issues, version drift, and supply chain vulnerabilities. This approach promises to streamline collaboration, enhance reliability, and significantly improve the security posture of cloud-native applications from inception to operation.

Capili explicitly positions this work as R&D, exploring the potential of fusing these two powerful paradigms. While acknowledging the steep learning curve often associated with Nix, the tutorial emphasizes Flux as an accessible entry point, allowing teams to leverage Nix's benefits without deep functional programming expertise. The ultimate goal is to shift GitOps further left, making declarative dependency management an inherent part of the daily developer experience, thereby ensuring that every change, from local development to production deployment, is version-controlled, reproducible, and auditable.

Background

▶ Watch: Introduction to Nix, GitOps, and workshop goals (0:00)

The problem of dependency management and environmental consistency has plagued software development for decades. Developers often face challenges when their local environment differs from that of a teammate, a continuous integration (CI) server, or the production environment. This leads to issues like "works on my machine," unexpected build failures, or even runtime errors in production due to subtle version discrepancies or missing libraries.

Traditional package managers (e.g., Homebrew on macOS, apt on Ubuntu, pacman on Arch Linux) are often system-wide and can lead to conflicts or different versions of the same package being installed across different machines or even for different projects on the same machine. This problem is exacerbated when teams use diverse operating systems and hardware architectures.

Docker emerged as a significant solution, abstracting away the underlying operating system and hardware by allowing applications and their dependencies to run in isolated Linux containers. While Docker significantly improved portability and consistency by "pretending everything was a Linux machine," it introduced its own trade-offs. These include the overhead of running virtual machines, splitting resources, and potentially not leveraging the benefits of native operating system kernels or specialized hardware (e.g., GPUs for ML workloads).

Nix (and the NixOS project) has been addressing these challenges for over 22 years. It is a functional package manager that enables the creation of reproducible software in a hermetic way. Nix builds software in isolated environments, ensuring that all dependencies, including compilers, libraries, and runtime components, are explicitly declared and version-locked. Each package or component in Nix is identified by a cryptographic hash that uniquely represents its entire dependency closure, stored in the Nix store. This content-addressable storage ensures that if any input changes, a new hash is generated, guaranteeing immutability and preventing accidental version drift. Nix's declarative nature allows developers to specify exactly what software they need, and Nix will build or fetch it in a consistent manner, regardless of the underlying system. However, Nix itself is built on a functional programming language, which can have a steep learning curve for newcomers.

GitOps is a paradigm for managing infrastructure and applications using Git as the single source of truth. It extends declarative infrastructure (like Kubernetes YAML manifests) with version control, automated reconciliation, and pull request-based workflows. Tools like Flux and Argo CD continuously monitor Git repositories for desired state changes and automatically apply them to the cluster, ensuring that the deployed state always matches the declared state in Git. While GitOps has proven highly effective for operational aspects, its application typically begins after development and artifact creation, leaving the developer's local environment outside its direct scope.

The talk proposes bridging this gap by using Flux (the product by Flocks, not the Flux CD project), an ease-of-use layer on top of Nix. Flux aims to simplify Nix adoption by providing a more familiar command-line interface and declarative environment definitions, making it easier for teams to share and manage Nix-based development environments. The integration seeks to bring the benefits of Nix's reproducibility and hermetic builds into the collaborative, declarative framework of GitOps, extending its reach from production infrastructure back to the very first lines of code.

Key Findings

▶ Watch: Instructions for setting up GitHub Code Spaces (3:50)

The central contribution of this talk is the demonstration of a cohesive workflow that extends GitOps principles to the entire software development lifecycle, starting from the developer's local machine, through CI/CD, and into Kubernetes production environments, all powered by Nix's reproducible builds via the Flux environment abstraction.

1. Declarative and Reproducible Development Environments: The primary finding is that Flux environments, built on Nix, allow developers to define all project dependencies (Go toolchain, C compilers, database clients, etc.) in a declarative manifest.toml file within the project repository. This file, along with its corresponding lock file, ensures that every team member, regardless of their operating system (e.g., macOS ARM64 or Linux AMD64) or local configuration, activates an identical, version-locked environment. This eliminates "works on my machine" issues by guaranteeing that the development environment is as consistent as the production environment.

1. Cross-Architecture and Cross-Platform Consistency: Flux environments leverage Nix's ability to cross-build packages for different architectures and operating systems. The demo highlighted that a single manifest.toml and lock file can define an environment that works identically on a Linux AMD64 Codespace and a macOS ARM64 (Apple Silicon) machine. When flux activate is run, Flux intelligently resolves and fetches the correct, pre-built binaries from the Nix store for the specific host architecture, ensuring environmental parity across diverse development setups.

1. Enhanced Software Supply Chain Security: By using content-addressed packages within the Nix store, the system inherently provides strong provenance for all dependencies. Every binary and library is uniquely identified by a cryptographic hash of its inputs, making it impossible for a dependency to silently change. This foundation lays the groundwork for robust Software Bill of Materials (SBOM) generation, allowing developers to query all transitive dependencies for a given environment or build artifact, significantly improving visibility and control over the software supply chain. While acknowledging current weaknesses in vulnerability correlation, the underlying mechanism is ripe for future integration.

1. Simplified Container Image Creation: Flux environments simplify the creation of minimal, reproducible container images. The flux containerize command can generate a base image containing only the environment's dependencies, built specifically for Linux. This base image can then be used in a multi-stage Docker build to add the application binary, resulting in a lean, production-ready container that inherits the reproducibility guarantees of Nix. This approach ensures that the container's runtime environment precisely matches the development and build environments, preventing build-time/runtime discrepancies.

1. Extending GitOps Leftward: The overarching finding is the feasibility and benefits of "shifting GitOps left." By making development environment definitions part of the Git repository and managing them declaratively, the collaborative and reconciliation mechanisms of GitOps are applied to developer tooling. Any change to dependencies becomes a version-controlled commit, visible in Git history, and automatically propagated to all users, mirroring the way infrastructure changes are managed in GitOps. This creates a unified, end-to-end declarative workflow from developer workstation to Kubernetes cluster.

Technical Deep Dive

▶ Watch: Exploring the dev container: Nix, Flux, and Kubernetes tools (5:50)

The technical demonstration centered around setting up a reproducible development environment for a Go application that interacts with a PostgreSQL database and uses ImageMagick C bindings. The core tools and concepts include GitHub Codespaces, Flux environments, the Nix store, and custom Dockerfile strategies for containerization.

Development Environment Setup

The tutorial began by guiding participants to set up a GitHub Codespace. This provides a consistent, cloud-based development environment. The Codespace environment itself was configured using a dev container that leverages Nix via Flux. This devcontainer.json configuration declaratively specifies tools like Docker, kubectl, Kind (Kubernetes in Docker), Helm, jq, and gnupg, ensuring they are all installed consistently from the Nix store.

Participants were instructed to clone two repositories:

1. github.com/ybox/cryo-en: An experimental repository for the Codespace setup.
2. github.com/stealthy/scale-nix-gitops: The main repository containing the demo application and Kubernetes manifests.

Crucially, both repositories needed to be added to the Codespace workspace using Command+Shift+P -> "Add folder to workspace" to prevent loss of work and ensure proper environment activation across different project contexts.

Flux Environments

The cornerstone of the reproducible development experience is the Flux environment. When a user navigates into a project directory (e.g., scale-nix-gitops/scrapbook-dev) and runs flux activate, they enter a subshell where all the project's declared dependencies are available.

• Declarative Manifests: Each Flux environment is defined by a manifest.toml file, typically located in a hidden .flux directory within the project. This file lists the desired packages and their versions. For the Go application, the scrapbook-dev environment included ca-certificates, gcc, go, imagemagick, and pkg-config.
• Lock Files: Alongside manifest.toml, Flux generates a lock file (similar to package-lock.json in npm or go.mod in Go). This lock file precisely pins the cryptographic hashes of all direct and transitive dependencies for all supported architectures (e.g., aarch64-darwin for Apple Silicon Macs and x86_64-linux for Codespaces/Linux machines). This ensures absolute reproducibility; activating the environment always yields the exact same software stack.
• Package Management Commands:
• flux install <package-name>: Adds a package to the current environment's manifest.toml and updates the lock file.
• flux search <keyword>: Searches the Flux package catalog (backed by Nix packages) for available software.
• flux show <package-name>: Displays available versions and supported CPU/OS architectures for a specific package.
• flux edit: Opens the manifest.toml in the default editor for manual modifications, performing validation on save.
• Multi-Architecture Support: A key benefit highlighted was Flux's ability to provide the correct binaries for the host system. When flux activate is run on a Mac, it pulls ARM64 Darwin binaries; on a Codespace (Linux AMD64), it pulls AMD64 Linux binaries, all from the same declarative manifest.toml and lock file. This is achieved by leveraging the vast, cross-built Nix packages repository, which contains over 120,000 packages.

Go Application Example

The demo application was a simple Go web server (main.go) built with the Go Fiber framework. It demonstrated several complex dependency scenarios:

• Connecting to a PostgreSQL database using a Go driver.
• Performing image manipulation using ImageMagick C bindings (go-graphics/imagick). This specifically highlighted the challenge of managing cross-language dependencies (Go calling C libraries), which Nix and Flux handle robustly. The gcc and imagemagick packages in the Flux environment provided the necessary C toolchain and libraries.

Local Service Management with Flux

Initially, the Go application failed to run because it couldn't connect to a PostgreSQL database. Capili intended to use flux activate --remote flocks/postgres --start-services to spin up a PostgreSQL instance directly from a remote Flux environment. Due to a momentary outage of the fluxhub service, this remote activation failed, showcasing the realities of live demos.

However, the workaround demonstrated the flexibility:

1. A new local directory (postgres) was created.
2. The manifest.toml for the PostgreSQL environment was manually copied from the flocks/flux-envs GitHub repository.
3. flux init was run in the new postgres directory, followed by pasting the manifest content.
4. flux activate --start-services was then used to initialize and run a PostgreSQL database locally. This command not only activated the PostgreSQL binaries but also started the database service, making it accessible on localhost.
5. Using psql, a table was imperatively created in the running database to support the Go application.

This process demonstrated how Flux environments can manage and even start services, providing a fully isolated and reproducible local development stack that includes databases and other backend services.

Containerization with Nix and Flux

The talk then moved to containerizing the application for deployment to Kubernetes. This involved creating Docker images that leverage the Nix-managed dependencies.

• flux containerize: This command takes the current Flux environment and builds a Docker image that contains only the dependencies, not the application code itself. This effectively creates a minimal base image. Capili explained this is an early prototype but shows the future potential. The process ensures that the Linux-native container image contains the exact same versions of dependencies as the development environment, including their specific build flags and hermetic properties.

• Custom Multi-Stage Dockerfile: For more fine-grained control and integration with existing Dockerfile-based workflows, a custom Dockerfile.fluxenv was presented, demonstrating a multi-stage build:

1. Builder Stage (FROM github.com/container-registry/flocks/flocks): This stage starts from a base image that includes Nix and Flux. The project's .flux environment directory (containing manifest.toml and the lock file) is copied into this stage.
2. Environment Activation: Inside the builder stage, flux activate is run. This ensures that all necessary dependencies are pulled into the Nix store within the container.
3. Nix Store Querying: A critical step involves querying the Nix store to identify all recursive runtime requisites for the activated environment. This is done using nix store --query --requisites and some shell scripting to gather all content-addressed paths. These paths are then hard-linked into a dedicated directory.
4. Runtime Stage (FROM alpine:latest or a minimal base): This stage starts with a lean base image (e.g., Alpine Linux). Only the hard-linked dependencies from the builder stage's dedicated directory are copied into the new image, specifically into the /nix/store path. This produces an extremely minimal image containing only the application's runtime dependencies, without Nix or Flux itself.

• Nix Wrapper/Loader Hacks: A common challenge when running Nix-built applications outside a Nix-managed system (like in a standard Docker container) is that they expect dependencies in the /nix/store and rely on specific environment variables (like LD_LIBRARY_PATH) or wrapper scripts to find them. The demo showed a "hacky" RUN command in the Dockerfile that invokes the activate script from the Flux environment within the container. This script sets up the necessary environment variables to inform the linker and loader where to find the C bindings and other dependencies. This ensures the Go application can correctly link against ImageMagick.

Kubernetes Deployment

Finally, the talk touched upon deploying the containerized application to Kubernetes using GitOps.

• Kubernetes Manifests: The scale-nix-gitops/cluster/app directory contained Kubernetes YAML manifests:
• db.yaml: Defines a PostgreSQL deployment (a single replica, not production-ready) and a service. It also includes a ConfigMap to initialize the database with a CREATE TABLE statement and a Secret to hold database credentials, which are then mounted as environment variables.
• scrapbook.yaml: Defines the Go application deployment and service. The deployment references the container image built earlier (e.g., scrapbook:latest) and includes an explicit args array to correctly invoke the Go binary within the Nix-activated container environment (e.g., ["/nix/store/...", "/path/to/go-image-app"]). It also passes database connection details via environment variables, aligning with the db.yaml service.
• Flux CD Integration: Capili briefly mentioned using the Flux operator (from Control Plane, his employer) to manage Flux CD instances themselves, allowing for auto-updating Flux deployments across a fleet of clusters. This further reinforces the GitOps principle by managing the GitOps toolchain itself declaratively. The cluster/flux-instance.yaml demonstrated how to define a Flux CD instance that points to the main Git repository, reconciling changes every minute. This means that the Git repository not only manages the application and database deployments but also the GitOps agent responsible for deploying them, creating a fully self-managing and self-updating GitOps setup.

Demo / Proof of Concept

▶ Watch: Cloning the secondary repository for micro-demos (8:00)

The entire presentation served as a live, interactive tutorial and proof of concept. Participants were guided through the following steps:

1. Codespace Initialization: Setting up a GitHub Codespace from the cryo-en repository, which pre-configured a development environment with Docker, Kubernetes tools, and Nix/Flux.
2. Repository Cloning and Workspace Setup: Cloning scale-nix-gitops and adding it to the Codespace workspace.
3. Go Development Environment Activation: Navigating to scale-nix-gitops/scrapbook-dev and running flux activate to obtain go, gcc, imagemagick, and other necessary development dependencies. The flux show command was used to inspect package versions and supported architectures.
4. Building the Go Application: Executing go build to compile the go-image-app binary, demonstrating the successful resolution of complex C bindings for ImageMagick within the Flux-managed environment.
5. Running the Go Application Locally (Initial Failure): Attempting to run go-image-app and observing the expected failure due to a missing PostgreSQL database connection.
6. PostgreSQL Environment Setup and Service Activation:

• Initially, an attempt was made to use flux activate --remote flocks/postgres --start-services. This failed due to an external fluxhub service outage, highlighting a real-world dependency challenge.
• As a workaround, the manifest.toml for the PostgreSQL environment was manually copied from the flocks/flux-envs GitHub repository into a local postgres folder.
• flux init and flux activate --start-services were then used to successfully spin up a local PostgreSQL database instance.

1. Database Initialization: Using psql (available from the activated PostgreSQL environment) to connect to the local database and imperatively create the necessary table for the Go application.
2. Successful Local Application Run: Rerunning go-image-app and confirming it could now connect to the database and serve the web interface, demonstrating the fully functional, locally reproducible development stack. The prompt displayed the stacked Flux environments (flux test, scrapbook-dev, postgres), indicating the active subshells.
3. Containerization (Base Image): Demonstrating flux containerize to create a Docker image containing only the development environment dependencies.
4. Custom Dockerfile for Minimal Runtime Image: Presenting a multi-stage Dockerfile (Dockerfile.fluxenv) that uses a Flux/Nix builder stage to extract essential runtime dependencies into a minimal Alpine-based image. This showcased how to achieve extremely small, reproducible container images without embedding the entire Nix system.
5. Application Containerization: Discussing a Dockerfile that performs a fully containerized Go build (using the Flux-generated base image), copies the application binary and web assets, and creates a final runtime image that includes only the ImageMagick C libraries needed for the Go application. This required explaining the "loader hack" to ensure the Go binary could find its Nix-managed C dependencies within the container.
6. Kubernetes Manifests Walkthrough: Reviewing the db.yaml and scrapbook.yaml for deploying the database and application to Kubernetes, emphasizing how environment variables and service names would be used for inter-service communication. The talk concluded before actually applying these manifests to a Kubernetes cluster due to time constraints, but the conceptual flow was clearly demonstrated.

The demo effectively showcased how Flux environments provide a robust, reproducible, and portable way to manage development dependencies and services, bridging the gap between local development and containerized deployments.

Defensive Implications

▶ Watch: Addressing dependency management and team collaboration challenges (8:40)

Integrating Nix and GitOps for development and deployment offers significant defensive advantages, primarily centered around reproducibility, supply chain integrity, and environmental consistency.

1. Enhanced Reproducibility and Attack Surface Reduction:

• Elimination of Environmental Drift: By ensuring that development, CI/CD, and production environments are built from the same declarative manifest.toml and locked dependencies, Nix eliminates version drift. This means that an attacker exploiting a vulnerability in a specific library version on a developer's machine cannot rely on that same vulnerability existing in production if the production environment is built from the same, locked definition.
• Hermetic Builds: Nix's hermetic builds guarantee that the build process is entirely isolated from the host system. This prevents unexpected dependencies from creeping into the build, reducing the potential for supply chain attacks where malicious code might be injected through compromised build tools or libraries not explicitly declared.
• Minimal Runtime Images: The ability to create extremely minimal container images containing only the necessary runtime dependencies (as demonstrated with the multi-stage Dockerfile) significantly reduces the attack surface. Unnecessary tools, libraries, or shells are excluded, limiting potential entry points for attackers.

1. Robust Software Supply Chain Security:

• Content-Addressed Storage and Provenance: The Nix store uses cryptographic hashes to identify every package and its entire dependency closure. This means that any change, no matter how small, results in a new hash. This provides an immutable, auditable record of every component used in a build, offering strong guarantees about the provenance of software artifacts.
• Foundation for SBOMs: The nix store --query --requisites command, as demonstrated, can generate a comprehensive list of all transitive dependencies for a given environment or artifact. This forms a powerful foundation for generating Software Bill of Materials (SBOMs), which are crucial for understanding and managing software risks. While the talk noted that a direct "SBOM flag" is not yet available, the underlying data is easily extractable. This allows organizations to know exactly what's in their software, enabling faster response to newly discovered vulnerabilities.
• Tamper Detection: Because every component is content-addressed, any unauthorized modification to a dependency (e.g., in a cache or registry) would result in a hash mismatch, immediately indicating tampering and preventing the compromised component from being used.

1. Improved Vulnerability Management (Potential):

• Targeted Patching: With precise knowledge of every dependency and its version, security teams can more accurately identify which applications are affected by a CVE and target patching efforts efficiently.
• Current Limitations: Capili explicitly acknowledged a current weakness in the Nix ecosystem regarding vulnerability correlation. There isn't a widely adopted, automated system for correlating CVEs with specific Nix packages. However, the foundational data (precise package versions and their hashes) exists, making this a solvable problem for future development or integration with external vulnerability databases.

1. Consistent Security Policy Enforcement:

• Declarative Security Tooling: Just as application dependencies are declared, security tools (linters, static analysis, vulnerability scanners) can also be included in Flux environments. This ensures that all developers and CI pipelines use the exact same versions of these tools, enforcing consistent security policies and checks across the board.
• Git-Driven Auditing: All changes to environment definitions, including security-related tools or dependency updates, are version-controlled in Git. This provides a clear audit trail, showing who made what changes and when, facilitating compliance and incident response.

By embracing Nix and GitOps, organizations can move towards a more secure software supply chain, where environmental consistency and artifact provenance are built-in features, not afterthoughts.

Key Takeaways

• Shift GitOps Left: Extend declarative, version-controlled dependency management from production infrastructure to the developer's local environment, using Git as the single source of truth for all software components.
• Achieve True Reproducibility: Leverage Nix's functional package management and content-addressed store (via Flux environments) to ensure identical development, build, and runtime environments across diverse operating systems and architectures.
• Simplify Cross-Platform Development: Flux environments automatically resolve and fetch correct binaries for different CPU/OS targets (e.g., ARM64 Mac, AMD64 Linux) from a single declarative definition, eliminating "works on my machine" issues.
• Enhance Software Supply Chain Security: Nix's hermetic builds and content-addressed packages provide strong provenance, enabling the generation of comprehensive SBOMs and making tampering immediately detectable.
• Streamline Containerization: Use flux containerize or custom multi-stage Dockerfiles to create minimal, reproducible container images that contain only the necessary, version-locked dependencies, reducing the attack surface.
• Collaborate with Confidence: Any dependency change, even if made imperatively during development, is captured declaratively in Git, ensuring all team members are always synchronized on the exact software stack.

About the Speaker(s)

Leigh Capili is a Staff Software Engineer at Control Plane. His work heavily involves the Flux project, a cloud-native GitOps continuous delivery solution, for which he serves as a maintainer. Capili is deeply engaged in research and development, particularly exploring novel approaches to software development and deployment in cloud-native environments. His expertise lies in GitOps, Kubernetes, and emerging technologies like Nix, aiming to bridge gaps in current developer workflows.

The talk was co-presented by Tanja Ulianova. The transcript does not provide specific details about Tanja Ulianova's professional background or affiliation, but she contributed to the presentation of this workshop.