A Fork to Reckon With: Minimizing Friction When Adopting... Alexander Perlman & Narayanamurthi Mari

A Fork to Reckon With: Minimizing Friction When Adopting Open Source Software

Speakers: Alexander Perlman, Machine Learning Platform Engineer, Capital One; Narayanamurthi Mari, Distinguished Engineer, Capital One

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=yeg-uoBYCO0

Overview

In today's fast-paced technological landscape, the adoption of open-source software (OSS) is almost a prerequisite for innovation and efficiency. However, for organizations operating in highly regulated industries, the path to leveraging off-the-shelf OSS is often fraught with challenges. This talk, "A Fork to Reckon With," delivered by Alexander Perlman and Narayanamurthi Mari from Capital One, addresses the pervasive "Not Invented Here" (NIH) syndrome by outlining practical strategies to bridge capability gaps in open-source dependencies, rather than resorting to costly and unsustainable custom builds.

The speakers, both seasoned platform engineers and open-source contributors, draw from their extensive experience at Capital One, a company navigating stringent security and regulatory requirements. They present four distinct approaches—contribution, forking, wrapping, and mutation—each with its own set of advantages and disadvantages, culminating in a detailed exploration of mutation using Kyverno as a powerful, low-friction solution for Kubernetes environments.

This article delves into the core problem of NIH, examines the nuances of each proposed solution, provides a technical deep dive into Kyverno's capabilities, and offers actionable defensive implications for organizations striving to embrace open-source innovation while maintaining compliance and operational excellence. The insights shared are particularly valuable for platform teams, security architects, and developers grappling with the complexities of integrating OSS into enterprise-grade, regulated environments.

Background

▶ Watch: Introduction and defining 'Not Invented Here' (NIH) (0:00)

The "Not Invented Here" (NIH) syndrome is a common pitfall in software development, characterized by organizations opting to build custom solutions from scratch rather than adopting mature, industry-standard, widely used open-source alternatives. Alexander Perlman vividly recounted an early career experience maintaining an in-house version of Terraform implemented in Ruby, a "nightmare" that solidified his conviction against NIH. While there are niche cases where building a bespoke solution might be justified, the speakers emphatically argue that the vast majority of the time, resisting the allure of NIH and leveraging existing OSS is the superior path. The benefits of OSS are undeniable: a distributed maintenance burden across a global network of engineers, broader impact, and the invaluable community collaboration that fosters learning and innovation. As an example, Perlman highlighted the staggering contributor count of projects like Kubernetes, illustrating the near impossibility for a single company to match such a collective effort.

However, the reality for companies in regulated industries, such as Capital One, often presents a significant hurdle: capability gaps. Off-the-shelf open-source software, while robust and feature-rich, frequently falls short of meeting specific internal mandates. These mandates can stem from elevated security requirements, strict regulatory compliance (e.g., data residency, audit logging, identity management), or unique business logic that is proprietary to the organization. When confronted with these gaps, organizations find themselves at a crossroads: either use the gaps as justification for reinventing the wheel (NIH) or strategically close those gaps within their open-source dependencies. The speakers firmly advocate for the latter, emphasizing that the effort required to close gaps in existing dependencies is almost always less than the monumental lift of building and maintaining a custom solution from the ground up.

The talk provided a non-exhaustive list of common capability gaps encountered, including the need for specific security contexts, enforcement of internal image repositories, resource quotas, network policies, and stringent audit logging—all requirements that standard OSS might not inherently fulfill without modification or augmentation. The subsequent sections of the talk then lay out a framework of four distinct approaches to tackle these challenges, moving from the most collaborative to the most introspective, with the ultimate goal of minimizing friction and maximizing the benefits of open-source adoption.

Key Findings

▶ Watch: The core challenge: addressing open source capability gaps (3:20)

The central finding of the talk is a comprehensive framework for addressing capability gaps in open-source software within regulated enterprise environments, offering alternatives to the detrimental "Not Invented Here" (NIH) syndrome. Perlman and Mari identified four primary strategies: Contribution, Forking, Wrapping, and Mutation. Each approach comes with a distinct set of trade-offs, making the selection highly dependent on the specific use case, urgency, and the nature of the required changes. The speakers emphasized that there is no single "one-size-fits-all" solution, and organizations may even transition between these strategies over time.

Contribution to the upstream open-source project is presented as the ideal solution, distributing maintenance burden and fostering community. However, its feasibility is limited by proprietary functionality that cannot be shared and the need for patience during the maintainer review and release cycles.

Forking offers total control and expedited delivery for urgent or proprietary changes, but at the significant cost of high maintenance burden, upgrade friction, and potential feature lag from the upstream.

Wrapping provides an abstraction layer over underlying dependencies, allowing for interface control and protection against volatility. Yet, it introduces substantial complexity, maintenance overhead, and the creation of "cursed knowledge" through proprietary interfaces that are not transferable to external roles.

The most novel and emphasized finding, particularly for Kubernetes environments, is Mutation. This approach, exemplified by the Kyverno policy engine, allows for the modification of Kubernetes manifests at runtime without altering the source code of the underlying applications. Mutation embraces a common, YAML-based approach, significantly reducing maintenance burden compared to forking or wrapping, and offering a powerful mechanism for enforcing enterprise-specific policies, security contexts, and resource management across a cluster.

Ultimately, the talk concludes that a strategic combination and thoughtful selection from these four methods, guided by a decision tree that considers factors like proprietary nature, urgency, and abstraction needs, is crucial for successful open-source adoption in regulated industries, enabling organizations to close capability gaps effectively while avoiding the pitfalls of NIH.

Technical Deep Dive

▶ Watch: Four approaches for capability gaps: contribute, fork, wrap, mutate (4:10)

The core of the talk revolved around a detailed examination of four distinct strategies for addressing capability gaps in open-source dependencies: Contribution, Forking, Wrapping, and Mutation.

Contribution

Contribution involves submitting changes directly to the upstream open-source project. This is the speakers' preferred method due to its inherent advantages.

• Pros: It distributes the maintenance burden across a global community of engineers, significantly reducing operational overhead for any single organization. It broadens the impact of the solution, potentially benefiting users worldwide, and fosters invaluable community engagement, learning, and collaboration. As Alexander Perlman noted, "Don't compete, collaborate."
• Cons: The primary limitation is that proprietary functionality cannot be contributed upstream. While pluggable architectures can sometimes mitigate this by allowing runtime injection of proprietary code, not all projects support this. Additionally, contribution requires patience, as organizations are at the mercy of maintainers' priorities for code review, merging, and release cycles, which can be an obstacle if immediate deployment is required.

Forking

A fork is a copy of an existing codebase where an organization applies its own changes. Notable examples include Valheim and OpenTofu. Capital One itself maintains an internal fork of the Kubeflow Pipelines UI.

• Pros: Forking grants total control over the codebase, removing reliance on upstream maintainers for reviews or releases. This allows for the incorporation of complex, proprietary business logic and significantly expedites delivery when urgency is a factor.
• Cons: The most significant drawback is a high maintenance burden. The organization becomes solely responsible for the forked code, leading to upgrade friction when attempting to rebase changes from the upstream origin, often resulting in merge conflicts. This can cause feature lag, where new functionalities in the upstream are delayed in reaching internal users.
• Mitigation: To minimize these disadvantages, speakers recommend keeping forks as small as possible in surface area. Forks can also serve as a staging ground for future open-source contributions; an urgent fix might start as a fork, then be contributed upstream, allowing the fork to be reduced or eliminated.

Wrapping

A wrapper abstracts underlying dependencies, either obfuscating them or providing a transparent layer. This can occur client-side or server-side. Kubeflow Pipelines, which wraps both Argo Workflows and Tekton to provide a machine learning-focused orchestration engine, serves as a prime example.

• Pros: Like forking, wrapping allows for the incorporation of complex business logic. It provides obfuscation of underlying dependencies, theoretically protecting against volatility (e.g., license changes) by allowing the offending dependency to be swapped out without impacting end-users. It also offers interface control, enabling the redefinition of user interfaces (API, CLI, GUI, SDK) to align with internal branding or conceptual frameworks.
• Cons: Wrappers share the same disadvantages as forks regarding high maintenance burden, upgrade friction, and feature lag. A significant amount of engineering time is often redirected from feature delivery to simply maintaining the wrapping logic, especially when targeting multiple complex dependencies. Wrappers introduce increased complexity, leading to brittleness and immense difficulty in debugging, as issues could originate in the end-user code, the wrapper, the underlying dependency, or even lower in the stack. Furthermore, proprietary interfaces created by wrappers can result in "cursed knowledge," where engineers develop skills that are not transferable to external roles, leading to frustration.

Mutation (with Kyverno)

Mutation is presented as the most novel and often preferred approach, especially in Kubernetes environments. It involves modifying Kubernetes manifests at runtime, or "on the fly," before they are created within the cluster.

• How it Works: When a user or application submits a manifest, the Kubernetes API server sends an HTTP call to configured admission webhooks. These webhooks, if they match the target, can modify the manifest before sending it back to the API server for creation. This means the source code of the open-source application itself remains untouched.
• Kyverno: The talk focuses on Kyverno, an open-source, general-purpose admission controller and YAML-based policy engine. It is a CNCF Incubation project (recorded in 2022) that simplifies policy enforcement without requiring users to learn specialized languages like Rego (used by OPA).
• Capabilities: Kyverno can validate manifests, mutate them, create (generate) new resources, delete resources, and verify images (beta feature for image exams). It can interact with the Kubernetes API server to fetch additional values for policy logic.
• Policies and Rules: Kyverno operates on policies, which can be cluster-level (applicable across all namespaces) or namespace-specific. Each policy contains multiple rules that define specific actions (e.g., inject environment variables, inject volumes).
• Targeting: Rules include match and exclude sections to precisely define targets based on Kubernetes objects (pods, namespaces), names, labels, or annotations.
• Architecture: Kyverno's high-level architecture includes three main controllers: the Admission Controller (for new resource validation and mutation), the Background Controller (for creating resources and mutating existing objects), and the Cleanup Controller (for resource deletion).
• Kyverno Examples: The speakers provided numerous practical examples of Kyverno policies:
• Image Repository Enforcement: Mutating container images from public registries (e.g., docker.io) to internal Artifactory repositories, solving issues where Kubeflow Pipelines might default to public images.
• Security Context Enforcement: Automatically setting security contexts (e.g., runAsUser, allowPrivilegeEscalation: false) on pods, addressing hardcoded settings in upstream applications that conflict with enterprise policies.
• Custom Resource Mutation: Enforcing behaviors on custom resources, such as setting a maximumIdleTimeout: 1h on Dask jobs to save costs.
• Pod Protection: Injecting annotations (e.g., Carpenter annotations) into long-running jobs (notebooks, XGBoost) to protect them from disruption by cluster autoscalers.
• Sidecar Injection: Automatically injecting sidecar containers (for metrics, orchestration, storage) along with their configurations (environment variables, security contexts, volumes), with preconditions to apply only to new objects.
• Validation: Enforcing required labels (e.g., project, team) on namespaces in multi-tenant environments. Validating resource requests and limits, ensuring the difference ratio is within a specified percentage (e.g., 20%).
• Resource Creation (Generate Policy): Automatically creating default resources, such as a workspace volume in user namespaces, eliminating the need for wrapper logic.
• Resource Deletion (Cleanup Policy): Implementing garbage collection for stale resources (e.g., deleting volumes older than 30 days in dev environments) by injecting cleanup.kyverno.io/ttl labels via mutation.
• Kyverno Pros: It is "easy as hell" due to its YAML-based nature, requiring no new programming language. It incurs no maintenance burden on individual open-source projects, embracing a common, cluster-wide solution. It is powerful, capable of reading from the Kubernetes API server and namespaces, with hundreds of examples available.
• Kyverno Cons: It introduces computational overhead as it intercepts every API request, though Kyverno supports high availability and scaling to mitigate this. Mutation errors are reflected in the Kyverno controller logs, not the application containers, requiring platform administrators to debug. However, these cons are generally considered manageable compared to the benefits.

The talk concluded with a comparison matrix and a decision tree (29:00), emphasizing that organizations should move between solutions as needs evolve. A fork might be a temporary solution until a contribution can be merged, and Kyverno policies should be periodically reviewed to see if upstream changes or new contributions can obviate their necessity. The overarching message is to avoid NIH by strategically closing gaps.

Demo / Proof of Concept

▶ Watch: Contribution: preferred approach, benefits, and drawbacks (4:30)

While the talk did not feature a live, interactive demonstration of Kyverno in action, it provided numerous concrete examples of Kyverno policies in YAML format. These examples served as practical illustrations of how mutation policies can be structured to achieve various objectives, such as image repository enforcement, security context injection, sidecar injection, and resource lifecycle management. The speakers walked through the structure of these policies, explaining their matching rules, preconditions, and actions, effectively showcasing Kyverno's capabilities and ease of use in a declarative manner.

Defensive Implications

▶ Watch: Forking: gaining total control over the codebase (7:00)

The insights from "A Fork to Reckon With" offer critical defensive implications for organizations aiming to secure and manage their Kubernetes environments while embracing open-source software:

1. Prioritize Open Source and Avoid NIH: The fundamental defensive posture is to actively combat the "Not Invented Here" syndrome. Building custom solutions for capabilities already present in mature open-source projects introduces immense security and maintenance debt. Defenders should advocate for and enable the adoption of OSS wherever possible, leveraging the collective security expertise and rapid patching capabilities of global communities.

1. Strategic Gap Closure: Understand the four strategies (Contribution, Forking, Wrapping, Mutation) and their trade-offs. This allows security and platform teams to make informed decisions about how to address capability gaps without resorting to full-blown custom development.

• Contribution First: For security enhancements or compliance features that are generally applicable and not proprietary, contributing upstream is the most robust long-term defense. It ensures the feature benefits from community review, maintenance, and broad adoption, reducing the internal burden.
• Minimizing Forks: If a fork is unavoidable for urgent security fixes or proprietary integrations, keep it as small and targeted as possible. Implement robust processes for rebasing from upstream and prioritize contributing those changes back to the main project to reduce the attack surface and maintenance overhead of divergent codebases.
• Cautious Wrapping: Wrappers can provide a strong abstraction layer for security, allowing underlying dependencies to be swapped out if vulnerabilities or license issues arise. However, the increased complexity and "cursed knowledge" can become a defensive liability, making debugging harder and potentially introducing new vulnerabilities in the wrapper itself. Use only when deep interface control or obfuscation is strictly necessary and the costs are justified.

1. Embrace Kubernetes Mutation with Kyverno for Policy Enforcement: For Kubernetes-native environments, Kyverno stands out as a powerful defensive tool.

• Centralized Policy Enforcement: Kyverno enables platform teams to centrally define and enforce security and compliance policies across all workloads without requiring application developers to modify their code. This includes:
• Image Security: Mutating container image references to internal, trusted registries (e.g., Artifactory) and validating image signatures to prevent the deployment of untrusted or vulnerable images.
• Security Contexts: Enforcing strict security contexts (e.g., runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, specific runAsUser and fsGroup) on all pods, ensuring least privilege principles are applied consistently.
• Resource Governance: Validating resource requests and limits (e.g., enforcing a 20% limit/request ratio) to prevent resource exhaustion attacks or inefficient resource utilization that could impact cluster stability.
• Network Policies: Automatically generating or validating network policies to enforce micro-segmentation and restrict communication between workloads.
• Sidecar Injection for Security: Injecting security-related sidecars (e.g., for logging, metrics collection, secrets management, service mesh proxies like Istio) into pods, ensuring consistent security tooling across the cluster.
• Label and Annotation Enforcement: Validating and mutating labels and annotations to enforce metadata for auditing, cost allocation, and operational management (e.g., ensuring all namespaces have project and team labels).
• Automated Remediation and Cleanup: Kyverno's ability to generate and delete resources can be leveraged for defensive purposes, such as automatically creating default secure configurations or cleaning up stale, unmanaged resources (e.g., old volumes, load balancers) that could pose a security risk or incur unnecessary costs.
• Compliance Automation: By defining policies as code, organizations can automate compliance checks and enforce regulatory requirements directly within their Kubernetes clusters, providing an auditable trail of policy application.

1. Operational Considerations for Kyverno:

• High Availability and Performance: Deploy Kyverno in a highly available configuration with sufficient replicas and bursting capacity to handle the computational overhead of intercepting API requests, ensuring it doesn't become a single point of failure or performance bottleneck.
• Monitoring and Alerting: Implement robust monitoring and alerting for the Kyverno controller itself. Since mutation errors manifest in the controller logs, platform administrators must be promptly notified to debug and resolve issues that prevent valid workloads from being deployed.
• Policy Management Lifecycle: Establish a clear lifecycle for Kyverno policies, including version control, testing (e.g., with kyverno test), and a review process. Regularly review policies to ensure they remain relevant and to identify opportunities to deprecate them if upstream projects integrate the desired functionality.

By strategically applying these defensive implications, organizations can navigate the complexities of open-source adoption in regulated environments, building secure, compliant, and efficient Kubernetes platforms that leverage the best of community-driven innovation.

Key Takeaways

• Avoid "Not Invented Here" (NIH) Syndrome: Resisting the urge to build custom solutions and instead leveraging mature, industry-standard open-source software is crucial for distributing maintenance burden, broadening impact, and fostering community collaboration.
• Contribution is the Ideal Solution: Whenever feasible, contributing directly to upstream open-source projects is the most effective way to close capability gaps, provided the functionality is not proprietary and there is patience for the maintainer review and release process.
• Kyverno-Based Mutation Offers Low-Friction Policy Enforcement: For Kubernetes environments, Kyverno is a powerful, YAML-based admission controller that enables runtime modification of manifests without touching application source code. It's excellent for enforcing security contexts, image repository policies, resource limits, and injecting sidecars, significantly reducing maintenance overhead.
• Forking and Wrapping Have Significant Downsides: While providing control and abstraction, forking leads to high maintenance burden and feature lag, and wrapping introduces complexity, debugging difficulties, and "cursed knowledge" due to proprietary interfaces. Use these approaches sparingly and strategically.
• No Single Solution Fits All: The choice between contribution, forking, wrapping, and mutation depends on factors like urgency, the proprietary nature of the change, and the need for abstraction. Organizations should be prepared to transition between these strategies as their needs and the upstream projects evolve.
• Continuously Evaluate and Refine: Regularly review existing internal solutions (forks, wrappers, Kyverno policies) to see if upstream projects have integrated the required capabilities, allowing for the deprecation of internal efforts and further embracing the open-source ecosystem.

About the Speaker(s)

Alexander Perlman is a Machine Learning Platform Engineer at Capital One, where he helps maintain an internal machine learning platform. He is also a Kubeflow member and contributor, demonstrating his active engagement with the open-source community. Perlman's professional experience has been shaped by a strong aversion to proprietary in-house solutions, advocating passionately for open-source adoption.

Narayanamurthi Mari (Morty) serves as a Distinguished Engineer at Capital One. He is deeply passionate about open-source Kubernetes, building platforms, and related services. Morty brings extensive expertise in platform engineering and distributed systems, contributing to the strategic direction and implementation of robust, scalable infrastructure. He resides in New Jersey with his wife and two children.