Workload Identity for Humans: A Twelve-Factor Approach - Vish Abrams, Heroku

Workload Identity for Humans: A Twelve-Factor Approach

Speakers: Vish Abrams, Chief Architect, Heroku

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=dNb1m84Bp4c

Overview

In this insightful KubeCon EU talk, Vish Abrams, Chief Architect at Heroku, tackles a pervasive and costly security challenge in modern application development: the management of workload identities and secrets. The presentation, titled "Workload Identity for Humans: A Twelve-Factor Approach," advocates for a fundamental shift away from the traditional practice of embedding long-lived secrets in application configuration. Abrams proposes a standardized, platform-managed system for generating and distributing short-lived, connection-scoped credentials, drawing inspiration from and aiming to extend the principles of the venerable 12-Factor App manifesto.

The core problem addressed is the inherent risk and operational burden associated with static, long-lived secrets – such as database passwords or API keys – that are typically stored as environment variables. These secrets are prone to accidental exposure, difficult to rotate, and can lead to severe security incidents. Abrams argues that while solutions exist, the lack of a universally adopted standard for workload identity makes it a "little bit hard" problem that every company solves independently, incurring immense aggregate cost and inconsistent security postures across the industry. This talk outlines a path toward a more secure, portable, and developer-friendly future where the platform, not the application developer, shoulders the complexity of credential management.

Background

▶ Watch: Introduction and defining workload identity (0:00)

The foundation of Abrams' discussion lies in the 12-Factor App manifesto, a set of principles for building robust, scalable applications that are portable across platforms. Originally published over a decade ago and recently open-sourced and updated by Heroku in collaboration with the community, the 12-Factor App defines a critical contract or interface between an application developer and a platform developer. This contract ensures that applications can be built to run consistently across various environments, from local development to staging and production.

A prime example of a successfully standardized interface is TLS termination. Most 12-factor applications don't concern themselves with TLS; they simply listen on a port, and the platform handles secure traffic routing. This abstraction allows developers to focus on application logic. However, not all interface definitions are equally mature. Abrams highlights the inconsistency in how applications handle domain names for OAuth callbacks, often resorting to manual configuration in environment variables. This pattern of relying on environment variables extends to how applications connect to backing services like databases, caches, and backend APIs. Developers typically store URIs and associated credentials (e.g., database passwords) as configuration variables. While this approach offers portability by allowing different URIs for staging versus production, it introduces a significant security vulnerability when these configuration variables contain sensitive, long-lived secrets.

Abrams details several critical problems with this prevalent practice:

1. Long-Lived Secrets: Configuration variables often persist for the entire lifetime of an application instance, meaning secrets can remain unchanged for weeks or months between deployments. This maximizes the window of vulnerability if a secret is compromised.
2. Synchronization Difficulty: Rotating a secret (e.g., a database password) requires careful orchestration between the service provider, the platform, and the application. A leak necessitates a complex, multi-week rotation process across potentially hundreds or thousands of microservices, creating a "not very fun time for the security team."
3. Accidental Exposure: Environment variables are frequently logged, cached, or inadvertently exposed in various debugging outputs or misconfigurations. This increases the likelihood of a secret leak, leading to cascading failures where exposed secrets must be rotated, exacerbating the synchronization challenge.

The inherent security risks and operational overhead of managing long-lived secrets within application configurations underscore the urgent need for a more robust and standardized approach to workload identity.

Key Findings

▶ Watch: The problem with secrets in configuration variables (4:00)

The central premise of Abrams' talk is a proposed new paradigm for workload identity, built upon four core tenets: short-lived, connection-scoped, platform-managed, and standardized credentials. These principles aim to fundamentally address the security and operational challenges posed by traditional secret management.

1. Short-Lived Credentials: The goal is for credentials to rotate frequently, ideally every 30-60 minutes. This drastically minimizes the window of opportunity for an attacker to exploit a compromised credential, significantly reducing the risk of exposure.
2. Connection-Scoped Identity: Instead of a single credential granting an application broad access to multiple services, each connection to a specific backing service receives its own unique credential. This prevents the confused deputy problem, where a service (Service B) could use a credential intended for it by another service (Service A) to impersonate Service A when interacting with a third service (Service C). By limiting the scope of each credential, a breach of one credential does not compromise access to all other services.
3. Platform-Managed Complexity: The platform, rather than the application developer, assumes responsibility for the complex lifecycle of these credentials. This includes generation, rotation, distribution, and validation. This abstraction allows developers to focus on writing application code, freeing them from the intricate security engineering required for robust credential management.
4. Standardized Interface: The ultimate objective is a universally adopted standard that ensures portability across different platforms and environments. This standardization leverages existing industry protocols like OIDC (OpenID Connect) and X.509 certificates, avoiding the pitfalls of "building your own crypto."

To overcome the chicken-and-egg problem of adoption – where applications need platform support, and platforms need application demand – Abrams introduces the concept of a CLI polyfill. This factor CLI acts as a local platform, enabling developers to integrate the proposed workload identity system in their local environments even before native platform support is widespread. This approach allows for early adoption and community collaboration on the standard.

Finally, a crucial underlying principle is the clear separation of authentication from authorization. The proposed system focuses on securely authenticating the identity of a workload. Once authenticated, existing policy enforcement mechanisms, such as Open Policy Agent (OPA), can be used to determine what actions that authenticated workload is authorized to perform.

Technical Deep Dive

▶ Watch: Proposing better workload identity: short-lived, platform-managed (6:00)

The technical architecture proposed by Abrams systematically addresses the challenges of secure workload identity. The approach begins with standardizing how applications discover and utilize credentials, then shifts the burden of secret management and validation to the platform.

The first step is to standardize credential storage in the environment. This means defining a consistent way for applications to receive connection information, including the URI of the service they need to connect to, and a pointer to the associated credentials. These credentials could be of various types, such as a username/password, a generic secret, or, ideally, a short-lived token.

Next, the platform takes on the critical role of synchronizing secrets between frontend and backend services. This ensures that when a secret is rotated, the platform updates all necessary components simultaneously, reducing the window of vulnerability and the operational burden on developers.

The most significant technical shift involves replacing traditional username/password secrets with short-lived, token-based credentials. Abrams emphasizes leveraging existing, well-vetted standards rather than inventing new cryptographic primitives. The primary candidates are:

• OIDC (OpenID Connect) based JWTs (JSON Web Tokens): A widely adopted standard for identity layer on top of OAuth 2.0.
• X.509 Certificates: Used for certificate-based authentication.

In the context of OIDC, the platform would provide the JWT to the application at a known, accessible location (e.g., a file path referenced in an environment variable). The application would then be responsible for loading this token and including it in an Authorization header when making requests to backend services. Concurrently, the platform would provide the backend service with the necessary information (e.g., public keys, issuer URLs) to validate these incoming tokens.

However, Abrams acknowledges significant challenges with direct application-level OIDC validation:

• Complexity: Validating OIDC tokens is non-trivial. Applications must handle token expiration, download and cache JSON Web Key Sets (JWKS), manage network outages, and ensure clock synchronization – tasks that are error-prone and divert developer focus from core application logic.
• X.509 and Mutual TLS: While X.509 is a viable option, its most common use for inter-application identity involves mutual TLS. This often requires the backend application to terminate the TLS connection itself. In many modern platforms, TLS termination is handled by the platform (e.g., an ingress controller or load balancer), meaning the backend application never sees the raw TLS handshake and thus cannot perform mutual TLS.

To circumvent these complexities, the proposed solution involves proxying incoming REST requests. When an application needs to connect to a backend service, the platform interposes a proxy in front of the backend. This proxy, which might be an existing component already handling TLS termination, is responsible for:

1. Validating incoming credentials: The proxy intercepts requests, extracts the OIDC JWT from the Authorization header, and performs all the necessary validation steps (expiration, signature verification against JWKS, issuer checks).
2. Providing an identifier to the backend: Upon successful validation, the proxy injects a trusted identifier into the request header (e.g., X-Client-ID) before forwarding it to the backend application. The backend application then simply trusts this header, knowing it has been validated by the platform.

On the sending side (client application), the process is streamlined:

1. Read the service URI from an environment variable.
2. Locate the associated credential (e.g., file path to a token).
3. Load the token from the specified location.
4. Include the token in the Authorization header of outgoing requests.
5. Token Refresh: Since tokens are short-lived, the application must refresh them. Common strategies include:

• inotify watching: The most compatible method, where the application monitors the token file for updates and reloads it when changed.
• On-demand reload: For infrequent requests, reloading the token before each request.
• Proactive expiry check: Parsing the OIDC token to predict upcoming expiration and refresh proactively.
• Retry on 403: A fallback where a 403 Unauthorized response triggers a token refresh and retry.

On the receiving side (backend application), the process becomes significantly simpler:

1. Check for the presence of the X-Client-ID header.
2. Trust the value of X-Client-ID as the authenticated identity of the caller, as it has been validated by the platform's proxy.

Abrams explicitly warns against outgoing proxies for workload identity, describing them as "dragons." Such proxies would act as a man-in-the-middle for all outgoing requests, requiring the proxy to re-sign SSL certificates for external endpoints (e.g., google.com), which is impractical and introduces significant security and operational overhead. Additionally, they can hide network retries and introduce unexpected latency, complicating application behavior. The proposed model focuses on incoming proxying, which aligns better with existing platform architectures.

To address the chicken-and-egg problem of adoption, where platforms await application demand and vice-versa, Abrams introduces the factor CLI as a polyfill. This command-line tool allows developers to run their applications locally, providing a simulated platform environment that handles workload identity. This means developers can immediately start building applications with the new identity model, even before their target cloud platforms natively support it. The factor CLI can:

• Manage local OIDC tokens.
• Integrate with Auth0 for identity.
• Utilize Kubernetes service accounts.
• Perform incoming credential validation via its own proxy.
• Automatically load .env files for configuration.
• Integrate with ngrok to expose local applications to the internet, facilitating remote callback validation for OIDC.

The vision extends to connecting with cloud services. Most major cloud providers (AWS S3, Google Cloud Storage, Azure Blob Storage) support OIDC tokens for access. However, setup remains complex, often requiring manual configuration in IAM consoles and tricky SDK integrations. Abrams suggests this is an area ripe for further standardization and automation. Similarly, non-HTTP services like PostgreSQL can be configured to validate OIDC tokens but typically require "janky" solutions like PAM plugins, highlighting a need for native support or specialized proxies for these protocols.

Finally, while this system provides robust authentication, authorization remains a separate concern. Abrams suggests using established tools like Open Policy Agent (OPA) to define and enforce policies based on the authenticated X-Client-ID within the application or via a dedicated authorization service. Looking to the future, the cloudpipe prototype specification, also in the 12-factor repository, aims to standardize APIs for inter-platform communication, simplifying the setup of cross-platform workload identity and automating complex cloud service integrations.

Demo / Proof of Concept

▶ Watch: Understanding connection-scoped identity to prevent abuse (6:28)

Vish Abrams presented a practical demonstration using a simple frontend and backend application to illustrate the factor CLI's capabilities and the proposed workload identity flow. The demo showcased the transition from insecure manual authentication to platform-managed, token-based identity, both locally and in a deployed environment.

Initially, the frontend and backend applications were run without the factor CLI. When the frontend made a request to the backend without an X-Client-ID header, the backend correctly returned unauthorized. However, if the frontend manually injected an X-Client-ID header, the backend would authorize the request, highlighting a security flaw: any client could impersonate another by simply providing the header.

The core of the demo then involved integrating the factor CLI polyfill:

1. Backend with factor CLI: The backend application was started using the factor CLI. This introduced a proxy layer in front of the backend. Now, when the frontend (still running without the CLI) sent a request, even with a manually injected X-Client-ID header, the backend received unauthorized. This demonstrated that the factor CLI's proxy was now intercepting the request and, since it wasn't validating a proper token, was not forwarding the X-Client-ID to the backend. The backend itself no longer directly trusted client-provided headers.
2. Frontend with factor CLI: Next, the frontend application was also started using the factor CLI. In this setup, the frontend loaded a short-lived token provided by its factor CLI instance and included it in the Authorization header. This token was then validated by the factor CLI proxy in front of the backend, which, upon successful validation, injected the trusted X-Client-ID header. The backend then authorized the request. This showed a complete, secure flow where the application code changes were minimal: simply loading credentials from a known location (a file watched by the app) and adding them to the request header.

Abrams briefly showed the minimal code required for this integration: on the sending side, loading backend credentials, watching the token file for changes, and passing the token in every request's Authorization header. On the receiving side, the backend merely checked for X-Client-ID and returned unauthorized if absent.

The demonstration then extended to a remote deployment scenario on Heroku. Abrams deployed the backend application to Heroku, utilizing a custom buildpack that wrapped the application in the factor CLI. This effectively deployed the polyfill remotely. The frontend application, still running locally (or also remotely with factor CLI), was configured with the Heroku-provided URI of the backend. An initial attempt at communication failed because, while the frontend was sending a token, the backend's factor CLI instance hadn't been configured with the necessary client credentials to validate that token. After setting a configuration option on the backend application to inform its factor CLI instance about the frontend's client credentials, the system functioned correctly, demonstrating the portability of the factor CLI approach to a cloud environment.

Finally, Abrams showcased the backend application connecting to an AWS S3 bucket. The backend, still managed by the factor CLI, was configured with credentials that allowed it to obtain an OIDC token. This token was then used by the AWS SDK (specifically, the web identity token file) to assume an IAM role in AWS, which was manually configured in the AWS IAM console to trust the OIDC issuer provided by the factor CLI. This allowed the backend to store incoming requests in S3 and retrieve them even after a restart, showcasing how platform-managed workload identity can simplify secure access to external cloud services with minimal developer code.

The demo effectively highlighted that the difference between running an application with and without the polyfill was minimal, typically involving only a few extra command-line words, yet it delivered significant security and operational benefits.

Defensive Implications

▶ Watch: First step: Standardizing credential storage in environment (7:15)

The adoption of a standardized, platform-managed workload identity system, as proposed by Vish Abrams, carries profound defensive implications for organizations. Moving away from the current paradigm of long-lived secrets embedded in configurations offers several critical improvements to an organization's security posture.

The most significant benefit is the elimination of long-lived secrets from application configurations. By transitioning to short-lived credentials that rotate frequently (e.g., every 30-60 minutes), the window of opportunity for an attacker to exploit a compromised secret is drastically reduced. Even if a token is exfiltrated, its utility to an attacker is severely limited by its short lifespan, making it far less valuable than a static, long-lived password.

Furthermore, the concept of connection-scoped credentials directly addresses the confused deputy problem. Each service-to-service interaction utilizes a unique credential, preventing an attacker from leveraging a compromised token to impersonate one service to gain unauthorized access to another. This significantly reduces the attack surface and limits the blast radius of a credential compromise.

The platform-managed nature of this system offloads the complexity of credential lifecycle management (generation, rotation, distribution, validation) from individual application teams to a centralized, specialized platform. This ensures that credential management is handled by security experts, reducing the likelihood of developer error or insecure custom implementations. It also greatly simplifies secret rotation, turning a complex, multi-week orchestration task into an automated, platform-handled process, thereby minimizing operational overhead and ensuring secrets are rotated promptly after a suspected breach.

The use of proxy-based validation for incoming requests enhances security by centralizing the intricate process of OIDC token validation. Instead of every backend application needing to correctly implement token validation (checking expiration, JWKS, signatures, clock sync, etc.), a trusted platform proxy handles this, injecting a verified identity (X-Client-ID) into the request. This reduces the burden on application developers and ensures consistent, robust validation across all services.

Standardization across the industry, driven by initiatives like the updated 12-factor manifesto, will lead to more secure and interoperable systems. By converging on a common approach, organizations can avoid the costly and error-prone cycle of each team or company inventing its own "little bit hard" solution, which often results in inconsistent security practices and vulnerabilities. The factor CLI polyfill is a critical defensive tool in this context, as it enables secure local development with identity, preventing developers from resorting to insecure practices (like disabling authentication) in their local environments.

From a policy enforcement perspective, the clear separation of authentication (who is this workload?) from authorization (what can this workload do?) is a strong defensive pattern. Once a workload is authenticated by the platform, organizations can leverage existing, powerful tools like Open Policy Agent (OPA) to define granular access policies based on the trusted X-Client-ID, ensuring that workloads only perform authorized actions.

However, defenders must also consider the implications and challenges of adopting this new model:

• Migration of Existing Applications: Integrating this new system into legacy applications that rely heavily on long-lived secrets will require a phased migration strategy.
• Non-HTTP Services: Securely extending this model to non-HTTP services (like databases) that currently lack native OIDC support will require specialized solutions or further standardization efforts.
• Cloud Service Integration: While cloud providers support OIDC, simplifying the complex IAM console configurations and inconsistent SDK support for web identity tokens is crucial for broader adoption and reduced misconfiguration risk.
• Platform Support: The full benefits are realized when the underlying platform natively supports this model, requiring investment and development from platform providers.

Ultimately, this approach represents a significant leap forward in workload security, shifting the responsibility for complex credential management to the platform and empowering developers to build more secure applications by default.

Key Takeaways

• Long-lived secrets in application configuration are a critical security and operational liability. They are prone to exposure, difficult to rotate, and lead to costly, cascading security incidents.
• Workload identity should be short-lived, connection-scoped, platform-managed, and standardized. This paradigm minimizes risk, prevents the confused deputy problem, offloads complexity from developers, and ensures portability across environments.
• Leverage existing standards like OIDC and X.509, but offload validation complexity to the platform. A trusted proxy handling incoming requests and injecting a verified identity (X-Client-ID) into headers simplifies security for backend applications.
• The factor CLI polyfill is essential to bridge the adoption gap. It allows developers to immediately implement and test the new workload identity model in local environments and even remotely, driving community feedback and accelerating platform adoption.
• Standardization across the industry is crucial. Addressing this "little bit hard" problem collectively will prevent every company from building its own costly, often inconsistent, and potentially insecure solutions.
• Collaboration is vital for refining and adopting this approach. The 12-factor manifesto is being updated, and community involvement is encouraged to shape the future of workload identity.

About the Speaker(s)

Vish Abrams is the Chief Architect at Heroku, a pioneering platform-as-a-service company. With his extensive experience in building and operating cloud platforms, Abrams is at the forefront of efforts to evolve fundamental principles of application development and deployment. He is actively involved in the open-sourcing and community-driven update of the 12-Factor App manifesto, seeking to integrate modern cloud-native concepts like workload identity into its foundational tenets. His work at Heroku includes developing internal implementations of workload identity solutions, which have successfully reduced reliance on static AWS credentials within their systems. Abrams is a strong advocate for standardization and collaboration within the cloud-native community to solve pervasive "little bit hard" problems that collectively create significant costs and security challenges for the industry.