Type-safe Feature Flagging in OpenFeature: Lessons Learned F... Michael Beemer & Florin-Mihai Anghel

Type-safe Feature Flagging in OpenFeature: Lessons Learned from Google

Speakers: Michael Beemer, Product Manager, DynaTrace; Florin-Mihai Anghel, Software Engineer, Google

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=mewXGSwDCE4

Overview

This talk, delivered by Florin-Mihai Anghel from Google and Michael Beemer from DynaTrace and OpenFeature, delves into the critical topic of type-safe feature flagging and the significant lessons learned from Google's extensive experience, culminating in the development of the OpenFeature CLI. Feature flags, while powerful tools for progressive rollouts, A/B testing, and rapid incident mitigation, introduce complexities that can lead to human errors and system outages. The speakers highlight how Google, with over a decade of using feature flags across its vast codebase, identified recurring issues stemming from inconsistencies between flag definitions in management systems and their usage in application code.

The core of their presentation revolves around a solution: code generation for type-safe flag accessors. This approach aims to eliminate common pitfalls such as misspelled flag names, mismatched default values, and forgotten cleanup of deprecated flags. By generating code that provides compile-time guarantees, developers can shift error detection left, preventing runtime issues and ensuring a smoother, more reliable feature rollout lifecycle. The talk culminates in a demonstration of the OpenFeature CLI, an open-source tool inspired by Google's internal practices, which brings these advanced type-safety capabilities to the broader developer community through the vendor-agnostic OpenFeature standard. This initiative is crucial for any organization seeking to enhance the reliability and efficiency of their feature flag implementations.

Background

▶ Watch: Introduction to type-safe feature flagging and agenda (0:00)

Feature flags represent a fundamental paradigm shift in how software is developed and deployed. At their core, they allow for the dynamic modification of a binary's behavior without requiring a new release. This capability enables developers to toggle features on or off, adjust functionality, or even change user experiences in real-time. The primary advantages include progressive rollouts, allowing new features to be gradually exposed to a subset of users, thereby reducing risk; rapid incident mitigation, as problematic features can be instantly disabled in production; and facilitating A/B testing and other forms of experimentation to gather user feedback and optimize product decisions.

Google's journey with feature flags dates back to 2009, making it a pioneer in their widespread adoption. Over the years, feature flags have become the de facto method for introducing new functionalities and releases across Google Cloud Platform. The scale of their usage is staggering: approximately 70% of Google developers utilize feature flags regularly, contributing to a codebase containing over 150,000 active feature flags. This immense scale, while demonstrating the utility of flags, also exposes their inherent challenges. One significant problem highlighted by Florin-Mihai Anghel is the cleanup of deprecated flags. Developers often introduce flags but neglect to remove them once the feature is fully rolled out and stabilized, leading to code bloat and technical debt. A humorous yet telling anecdote shared was YouTube's policy at one point: to introduce a new feature flag, a developer had to clean up two existing ones, underscoring the severity of this issue.

The talk meticulously dissects the challenges arising from the inherent multiple sources of truth in a typical feature flagging setup. A feature flag's definition resides in a centralized flag management system, while its usage is embedded within the application's source code. These two components operate on different release cycles: the flag service can be updated very frequently, propagating changes almost instantly, whereas the application might have a weekly or bi-weekly release schedule. This desynchronization creates a fertile ground for errors. For instance, a flag might exist in the application code but not yet in the flag service, or vice-versa.

The speakers provided concrete examples of outages caused by these inconsistencies, all stemming from human error:

• Trailing Whitespace: A developer accidentally added a trailing whitespace to a flag name, causing the application to query an non-existent flag. This resulted in the flag defaulting to an unexpected value, leading to a drop in feature usage.
• Default Value Mismatch: A mismatch between the default value defined in the code and the default in the flag management system led to a feature being fully launched overnight instead of a gradual rollout, causing an outage.
• Premature Flag Removal: A flag was removed from the flag service under the assumption that all code references had been cleaned up. However, residual code references defaulted to an unexpected value, effectively disabling a feature and causing another outage.

These incidents underscore a critical need: to move beyond mere carefulness and implement systemic solutions that prevent such errors from occurring in the first place. The overarching question posed by the speakers is, "How can we help as developers to solve the problem before we even need to check into the code of all the flag references, make sure all these things are in sync? It should just be smooth." This forms the bedrock for the subsequent discussion on type-safe code generation.

Key Findings

▶ Watch: Definition and advantages of feature flags (1:10)

The central finding and solution presented by the speakers, born from Google's extensive experience, is the implementation of code generation for type-safe flag accessors. This approach directly addresses the "multiple sources of truth" problem and the human errors that plague traditional feature flag implementations.

Google's internal solution, which inspired the open-source work, involves generating all flag accessors based on flag configurations defined in the flag management system. This offers several critical benefits:

• Mitigation of Mistakes: It entirely eliminates errors caused by typos or incorrect flag names, as developers interact with strongly typed methods rather than hardcoded strings.
• Direct Source of Truth: The generated code becomes the single, authoritative source for what flags are available and how they should be accessed within the binary. At Google, this was taken a step further by completely disallowing reflective access to flags, meaning the generated accessors are the only way to interact with flags.
• Exhaustive List and Validation: The generated code provides a complete and exhaustive list of all accessible flags, making it significantly easier to validate consistency between the application and the flag management system.
• Compile-Time Error for Stale References: When a flag is deprecated and removed from the flag management system, the subsequent code generation will no longer produce its accessor. Any remaining references in the application code will then cause a compile-time error, forcing developers to clean up stale code and preventing accidental runtime issues.

Recognizing the widespread applicability of these lessons, the speakers, particularly Michael Beemer, highlighted OpenFeature as the ideal open-source vehicle for bringing these best practices to the broader community. OpenFeature is a CNCF incubating project that provides an open specification for feature flagging. Its primary goal is to unify SDKs and standardize the way developers interact with feature flags, irrespective of the underlying vendor or homemade solution.

The advantages of standardizing with OpenFeature are threefold:

1. Avoid Vendor Lock-in: By abstracting away vendor-specific SDKs, OpenFeature allows organizations to switch flag providers without significant code changes, preserving architectural flexibility.
2. Community Building: It fosters a collaborative environment where developers can share experiences, use cases, and challenges related to feature flagging.
3. Shared Tooling: The community can collectively build and maintain high-quality tooling that benefits everyone, such as the OpenFeature CLI.

The OpenFeature CLI is a brand new initiative directly inspired by Google's codegen approach. It aims to drastically improve the developer experience with feature flags through three core components:

1. Vendor-Agnostic Flag Management System Hook: It can connect to any flag management system, including custom internal solutions, to retrieve flag definitions.
2. Local Manifest: It fetches flag definitions from the management system and stores them as a local JSON manifest within the repository. This manifest serves as a reliable, version-controlled reference for the flags.
3. Code Generation: Using the local manifest, the CLI generates type-safe flag accessors for various programming languages and frameworks.

This combination of Google's hard-won lessons, OpenFeature's standardization efforts, and the new OpenFeature CLI marks a significant step forward in making feature flagging more robust, reliable, and developer-friendly.

Technical Deep Dive

▶ Watch: Google's extensive use and cleanup challenges (2:30)

The technical architecture of OpenFeature is designed to provide a flexible and vendor-agnostic abstraction layer for feature flagging. As illustrated in the talk, the core components are the OpenFeature SDK within the application and the OpenFeature Provider, which acts as an interface to the actual flag management system. The flag management system itself can be any commercial product or a custom-built solution.

Prior to the OpenFeature CLI and code generation, interacting with feature flags typically involved hardcoded strings. A common Node.js example would look like this:

Here, 'with-cows' is a magic string. If this string is misspelled in the code or if the corresponding flag name changes in the flag management system, the getBooleanValue call would likely return the default false value without any compile-time warning, potentially leading to unexpected behavior in production.

The OpenFeature CLI revolutionizes this interaction by introducing a structured, automated workflow for type-safe flag accessors:

1. Feature Flag Creation: The process begins in the flag management system (e.g., a UI for product managers). A new feature flag is defined, for example, offer-free-shipping. This definition includes a unique key (offer-free-shipping), its expected type (e.g., boolean), and potentially a description and default value. This is the initial "source of truth" for the flag's existence and basic properties.

1. Fetching the Local Manifest: The OpenFeature CLI is then used to fetch the latest flag definitions from the flag management system. This command, (e.g., openfeature fetch), pulls the essential metadata for configured flags and stores it in a local manifest file, typically a flag.json file, within the application's repository. This manifest contains the flag key, its type, and crucial metadata such as the default value that should be used if the flag cannot be resolved (e.g., if the flag service is unavailable or the flag is not defined there). This flag.json file becomes a version-controlled, local source of truth for the expected behavior of flags in the application.

An example manifest entry might look like:

1. Code Generation: With the local manifest in place, the CLI's generate command (e.g., openfeature generate react or openfeature generate Node.js) is executed. This command reads the flag.json manifest and produces type-safe accessor code tailored to the target language and framework. For a React application, it might generate a custom React hook; for Node.js, it might generate a client method.

For the offer-free-shipping flag, the generated code might look like:

And the application code would then transform from:

to:

The impact of this seemingly small change is profound. The key technical advantages include:

• Elimination of Human Error: Typographical errors in flag names are caught at compile time, as the generated function useOfferFreeShipping() is a strongly typed entity. If a developer tries to call useOfferFreeShippin() (with a typo), the compiler will immediately flag it as an undefined function.
• Rich IDE Integrations: Modern Integrated Development Environments (IDEs) can leverage the generated code to provide autocompletion for flag accessors and display JSDocs (or equivalent in other languages) on hover. This means developers instantly see the flag's description, type, and default value without needing to consult external documentation or the flag management system.
• Compiler-Enforced Code Cleanup: As highlighted by Florin-Mihai, this is a critical benefit. When a feature flag is deprecated and removed from the flag management system, it is subsequently removed from the flag.json manifest during the next fetch operation. Consequently, the generate command will no longer produce its accessor function. Any remaining calls to the now-non-existent accessor in the application code will result in a compile-time error, forcing developers to clean up stale references before the application can be built. This mechanism actively combats the "developers don't like to clean up flags" problem.
• Enforced Consistency (Google's Approach): By disallowing reflective access to flags and making the generated code the only pathway, Google ensures that the binary's understanding of available flags is perfectly aligned with the definitions in the flag management system. This provides a high degree of confidence in the integrity of the feature flagging system.

The OpenFeature CLI, by externalizing Google's internal best practices into an open-source, vendor-agnostic tool, provides a robust solution for managing the complexities of feature flags at scale.

Demo / Proof of Concept

▶ Watch: Problem: Risk of flag name mismatches and accidental rollouts (4:20)

The speakers provided a live demonstration of the OpenFeature CLI's capabilities using a sample application called Toggle Shop, an e-commerce site for switches and toggles. The demo focused on a feature that displays a "Free shipping on orders over $50" banner.

1. Initial State (Hardcoded String): The banner's visibility was controlled by a feature flag accessed using the traditional hardcoded string method: client.getBooleanValue('offer-free-shipping', false). The banner was initially displayed, indicating the flag was enabled.

1. Demonstrating Human Error: Michael Beemer intentionally introduced a typo into the flag key in the application code, changing 'offer-free-shipping' to 'offer-free-shippin'. Upon refreshing the application, the banner disappeared. This vividly illustrated how a simple typo, easily missed in a large codebase, could lead to unexpected behavior in production, defaulting to false because the misspelled flag was not found. This emphasized the problem of relying on "good defaults" to mask fundamental errors.

1. Introducing the Local Manifest: The demo then showed a flag.json manifest file, which contained the correct definition for the offer-free-shipping flag, pulled from a hypothetical flag management system.

1. Code Generation in Action: Using the terminal, Michael executed the openfeature generate react command. The CLI processed the flag.json and generated a new TypeScript file containing a React hook named useOfferFreeShipping. The generated code included valuable JSDocs extracted from the flag's metadata in the manifest, providing a description and default value.

1. Adopting Type-Safe Accessors: The application's landing page code was then updated to replace the problematic client.getBooleanValue call with the newly generated useOfferFreeShipping() hook. As the speaker typed useOfferFreeShipping, the IDE's autocompletion immediately suggested the function, and hovering over it displayed the detailed JSDocs, showcasing the improved developer experience.

1. Restoring Functionality: After saving the changes, the banner instantly reappeared on the Toggle Shop, confirming that the type-safe accessor was correctly reading the feature flag's enabled state.

1. Live Toggling: To further prove the dynamic nature and correctness of the solution, Michael then toggled the offer-free-shipping flag off in the backend flag management system. Upon refreshing the Toggle Shop, the banner disappeared. He then re-enabled it, and the banner reappeared. This final step confirmed that the type-safe generated code seamlessly integrated with the dynamic nature of feature flags, providing both compile-time safety and runtime flexibility.

The demo effectively showcased how the OpenFeature CLI, by leveraging code generation from a local manifest, eliminates the risk of common human errors like typos, enhances developer productivity through IDE integration, and ensures that the application's understanding of feature flags remains perfectly synchronized with the flag management system.

Defensive Implications

▶ Watch: Identifying the 'multiple sources of truth' problem (5:40)

The insights and tooling presented in this talk offer critical defensive implications for any organization utilizing feature flags, aiming to enhance reliability, reduce outages, and streamline development workflows.

1. Adopt OpenFeature for Vendor Independence and Standardization: Defenders should strongly consider adopting the OpenFeature standard. This strategic move provides vendor independence, protecting organizations from being locked into proprietary feature flag solutions. Should a vendor's service become too expensive, unreliable, or cease to exist, migrating to another provider or an in-house solution becomes significantly less disruptive due to the unified SDK specification. This standardization also fosters a robust community and shared tooling, reducing the burden of developing and maintaining in-house solutions.

1. Implement Code Generation for Type Safety: The most significant defensive measure is to integrate code generation for feature flag accessors into the development pipeline. This directly addresses the root cause of many feature flag-related outages: human error, particularly typos in flag names. By generating strongly typed functions or hooks (e.g., useOfferFreeShipping() instead of client.getBooleanValue('offer-free-shipping', false)), developers gain compile-time guarantees. If a flag name is misspelled or changed, the compiler will flag the error immediately, shifting error detection left from runtime to build time.

1. Establish a Single Source of Truth with Flag Manifests: Utilize the OpenFeature CLI's capability to create a local flag manifest (e.g., flag.json) in the repository. This manifest, pulled directly from the flag management system, serves as a version-controlled, auditable single source of truth for all feature flag definitions accessible by the application. This ensures that developers are always working with the most current and accurate flag information, including types, default values, and descriptions.

1. Enforce Flag Definition Precedence: Emulate Google's policy: a flag must first be defined in the flag management system before it is allowed to exist in the host application's code. This prevents situations where code expects a flag that isn't yet configurable, leading to unexpected default behaviors. The code generation process naturally supports this by only generating accessors for flags present in the manifest, which in turn is sourced from the management system.

1. Leverage Compile-Time Checks for Deprecation and Cleanup: The code generation approach provides an invaluable defensive mechanism for flag lifecycle management. When a flag is deprecated and removed from the flag management system, its accessor will no longer be generated. Any lingering references in the application code will then cause a compile-time error. This actively forces developers to clean up stale flag references, preventing dead code, reducing technical debt, and eliminating the risk of accidental feature re-activation or unexpected defaults due to forgotten flags. This directly addresses the "developers don't like to clean up flags" problem.

1. Enhance Developer Experience to Prevent Errors: The rich IDE integrations (autocompletion, JSDocs on hover) provided by generated code are not just conveniences; they are defensive tools. By making it easier and more intuitive for developers to correctly use feature flags, the likelihood of errors is significantly reduced. Developers are less likely to make typos or misuse flags when they have immediate, in-context documentation and validation.

1. Promote a Culture of Flag Hygiene: While technical solutions are paramount, organizations should also cultivate a culture that prioritizes feature flag hygiene. Policies like YouTube's "clean up two flags for every one new flag" can be effective in preventing flag sprawl. The code generation mechanism supports this by making the cost of not cleaning up (i.e., compile errors) immediately visible.

By implementing these defensive strategies, organizations can transform feature flags from a potential source of outages and complexity into a robust, reliable, and developer-friendly mechanism for agile software delivery.

Key Takeaways

• Feature flags are powerful but prone to human error: While enabling progressive rollouts, A/B testing, and rapid incident response, traditional hardcoded string access to feature flags frequently leads to human errors like typos, name mismatches, and forgotten cleanup, which can cause significant outages.
• Type-safe code generation is the solution to common flag errors: Generating type-safe accessors from a centralized flag manifest eliminates the risk of misspelled flag names and ensures consistency between the flag management system and application code, catching errors at compile time rather than runtime.
• OpenFeature provides an open standard and tooling for robust flagging: As a CNCF incubating project, OpenFeature offers a vendor-agnostic specification that unifies SDKs across different providers, enabling organizations to avoid vendor lock-in and leverage community-driven, best-practice tooling like the OpenFeature CLI.
• Enforcing flag definition precedence prevents unexpected behavior: A critical best practice, learned from Google, is to ensure that a feature flag is fully defined and available in the flag management system before any code references it. The OpenFeature CLI's manifest and code generation workflow naturally support this by only generating accessors for existing, defined flags.
• Code generation aids in seamless flag deprecation and cleanup: When a flag is removed from the management system, its generated accessor disappears, causing compile-time errors for any remaining references in the application. This mechanism actively forces developers to clean up stale code, reducing technical debt and preventing accidental re-activation or misconfiguration of old features.
• The OpenFeature CLI is actively developing and seeking contributions: The OpenFeature CLI is a new, evolving tool that brings Google's lessons to the open-source community. Developers are encouraged to try it out, provide feedback on their user journey, and contribute by adding support for their favorite languages and frameworks.

About the Speaker(s)

Florin-Mihai Anghel is a Software Engineer at Google, where he has been working for three years. His work primarily focuses on feature flags and enhancing the safety of rollouts across the Google Cloud Platform. He is also an active open-source contributor to the OpenFeature project, bringing his extensive experience from Google's large-scale feature flag implementation to the community.

Michael Beemer is a Product Manager at DynaTrace, with a long and varied career in software development. He is a passionate and active open-source contributor, and notably, a co-founder of OpenFeature and a member of its governance committee. His involvement highlights his commitment to standardizing and improving the developer experience with feature flagging across the industry.