From Chaos To Control: Migrating Access Control... Jo Guerreiro & Poovamraj Thanganadar Thiagarajan

From Chaos To Control: Migrating Access Control... Jo Guerreiro & Poovamraj Thanganadar Thiagarajan

Speakers: Jo Guerreiro, Identity and Access Team, Grafana; Poovamraj Thanganadar Thiagarajan, Technical Lead, FGA Team

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=ZOG1J1Niuh0

Overview

This talk, "From Chaos To Control: Migrating Access Control," delves into the complex journey undertaken by Grafana to modernize its authorization infrastructure by migrating to OpenFGA, a declarative authorization system inspired by Google's Zanzibar. Presented by Jo Guerreiro and Poovamraj Thanganadar Thiagarajan, the session provides a candid look at the challenges and solutions encountered when transitioning a large, multi-tenant, cloud-native observability platform with over 14 years of legacy access control features. The speakers highlight the critical need for a standardized, fine-grained, and scalable authorization system in today's interconnected application ecosystems.

The core problem addressed by the speakers is the pervasive lack of standardization in authorization across the software industry, leading to bespoke, siloed implementations that introduce security risks and operational overhead. Grafana's experience serves as a real-world case study, offering invaluable insights for organizations grappling with similar legacy systems and the increasing demands for granular access control. The talk is particularly relevant for architects, security engineers, and developers working with multi-tenant cloud platforms, Kubernetes, and distributed systems, providing practical advice on schema design, data migration strategies, and performance considerations for authorization at scale.

Background

▶ Watch: Grafana's challenge: Migrating multi-tenant access control. (0:45)

The evolution of software development has seen significant standardization in many areas, such as authentication with protocols like OpenID Connect, OAuth, and SAML, and data storage with relational databases. However, authorization remains a fragmented landscape. As Poovamraj Thiagarajan points out, "Each product builds their own authorization. There is no standard we usually adopt. Sometimes even within the same product, each service does authorization differently." This lack of a unified approach creates significant problems: silos, specialized patterns, knowledge gaps, and severe security vulnerabilities, as evidenced by authorization-related issues consistently appearing at the top of the OWASP Top 10 and OWASP API Top 10 lists.

Historically, the industry faced a similar challenge with data storage before the advent of the relational model. Applications used custom logic for data storage and access, leading to tight coupling, redundancy, and inconsistency. The relational model provided the "right abstraction" that fit 90% of use cases, revolutionizing data management. In 2019, Google published a white paper titled "Zanzibar: Google’s Consistent, Global Authorization System," which proposed a generic, centralized authorization system used by hundreds of client services within Google, including Calendar, Cloud, Drive, Maps, Photos, and YouTube. Zanzibar demonstrated the feasibility of a unified access control system that could adapt to diverse application needs and even support complex features like search indexes that respect access control.

Inspired by Google's Zanzibar, OpenFGA emerged as a modular, open-source authorization system designed to provide a uniform data model and configuration language for expressing a wide range of access control policies. As a CNCF Sandbox project, OpenFGA aims to bring declarative authorization to the cloud-native ecosystem, offering first-class Kubernetes support. Its declarative nature, akin to popular technologies like SQL, Kubernetes manifests, Terraform, and React, promises to simplify the management of authorization for teams with hundreds or thousands of developers, ensuring consistency and central control.

Grafana's journey into OpenFGA began at KubeCon Paris in 2024, where they were introduced to the project. Their own system, starting as a monolith with haphazard access control, evolved into an interconnected observability platform supporting numerous applications (Kubernetes monitoring, synthetics monitoring, K6, IRM, SLO). This growth, coupled with the introduction of plugins and the need for fine-grained access control, highlighted the limitations of their existing, tightly integrated authorization system. The upcoming Grafana App Platform, designed to unify core and plugin features with shared interfaces and storage, served as the catalyst, making the evolution of their access control system from an option to a necessity.

Key Findings

▶ Watch: Unpacking why authorization is a uniquely hard problem. (3:35)

The talk reveals several key findings from Grafana's migration to OpenFGA, spanning both the capabilities of the authorization system and the practicalities of a large-scale enterprise transition.

Firstly, OpenFGA's declarative authorization model was a significant discovery. It allows for the definition of entities (e.g., users, teams, folders, dashboards) and their relationships (e.g., a user is a member of a team, a dashboard is in a folder). Permissions are then defined based on these relationships. The system's expressiveness allows for complex authorization rules, such as assigning a user as an admin of a folder, or making all members of a team a viewer. This includes powerful features like wildcards (e.g., user:* for public access) and algebraic operations (unions, intersections, negations) to combine relations. Crucially, inheritance can be modeled across multiple levels (e.g., a viewer of a parent folder is also a viewer of its child), simplifying the representation of hierarchical permissions. This declarative approach drastically reduces the lines of code needed to implement sophisticated authorization logic compared to imperative methods.

Secondly, Grafana identified specific needs driving their migration. Their platform required fine-grained access control beyond coarse-grained roles, especially as their observability platform became "extremely interconnected" with various applications talking to each other, not just to central Grafana. This distributed interaction meant that authorization checks could no longer solely reside within the core Grafana monolith. The Grafana App Platform initiative provided the opportune moment, mandating a new system that could support deep integration across all resources, whether core or plugin-based.

Thirdly, the migration uncovered critical challenges and lessons learned. Jo Guerreiro candidly shared that while OpenFGA offers immense power, practical considerations are paramount:

• Tuple Count: The number of tuples (permission statements) written directly impacts system performance. More tuples can lead to slower read operations, emphasizing the need for efficient schema design.
• Schema Complexity: While OpenFGA allows for highly expressive schemas, "schema complexity can actually affect your performance." Attempts to directly map all existing Grafana concepts into a complex OpenFGA schema proved detrimental. A simpler, more OpenFGA-native approach was often better.
• Database Performance Differences: OpenFGA supports various databases (MySQL, PostgreSQL, SQLite), but their performance characteristics are not uniform. Grafana, for instance, contributed the SQLite integration but noted that optimizations are still needed for certain databases like MySQL.
• Multi-tenancy Strategy: For Grafana's cloud-native, multi-tenant environment, the chosen approach was to assign each tenant its own dedicated store within OpenFGA. This provides strong isolation, preventing accidental cross-tenant permissions, and simplifying deployment by namespacing tenant information. Grafana runs "one OpenFGA per cluster" (per region), avoiding global latencies.
• Data Migration Lessons: Migrating legacy data is fraught with peril. Grafana learned to implement feature toggles that are "always write" to both old and new pathways to ensure behavior consistency, even if users toggle features on and off. They emphasized continuous verification of state between old and new stores, rather than relying on one-time migrations, especially given the strict consistency requirements of access control. Finally, they advocated for shadow calls, sending requests to both the legacy and new authorization engines simultaneously during rollout to validate responses and refine the schema before full cutover.

Finally, the talk highlighted the complexity of search with permissions. This "hard problem" requires different strategies depending on the scale of data and the restrictiveness of filters. Simple "search and check" (querying the database then checking each result with OpenFGA) works for small result sets. For medium scales (user can access < 1000 objects), filtering by access (asking OpenFGA what objects a user can access, then feeding those IDs to a SQL query) is effective. However, for high-cardinality access (more than 1000 objects), both methods break down. Local search indexes can pre-filter but introduce synchronization overhead. Grafana's pragmatic approach involves "aggressively filtering at the beginning" using tenant-specific information (e.g., stack ID from OpenID tokens) to reduce the dataset before applying authorization checks.

Technical Deep Dive

▶ Watch: Google's Zanzibar: Inspiration for a generic authorization system. (4:55)

The technical deep dive into Grafana's migration to OpenFGA reveals sophisticated architectural decisions and implementation details. At its core, OpenFGA leverages a declarative authorization model inspired by Zanzibar. This model defines relationships between entities and resources. For Grafana, entities include user, team, folder, and dashboard. Relationships are expressed as user is a member of team, dashboard is in folder, or user is an admin of folder. Permissions are then derived from these relationships.

For instance, to define that only admins can create a dashboard inside a folder, a simple line is added to the OpenFGA model. The power of OpenFGA comes from its ability to use algebraic operations and inheritance to define complex permissions concisely. A viewer of a folder might be defined as anyone explicitly assigned viewer or anyone who is an admin of that folder (a union operation). Further, a viewer of a parent folder can automatically be a viewer of a child folder, demonstrating multi-level inheritance. This flexibility allows Grafana to represent intricate authorization hierarchies spanning users, teams, organizations, and guest users without extensive custom code.

Grafana's approach to identity and authorization checks is two-fold. For user authorization, they rely on ID tokens from OpenID Connect, extracting two crucial pieces of information: the audience (representing the tenant ID or stack ID) and the sub (representing the user ID). These are then used to query OpenFGA, asking questions like: "Does user ID 31, part of tenant ID 3, have read permission on folder 'Airwin'?" For system authorization, which requires more coarse-grained access, Grafana uses traditional access tokens with defined scopes (e.g., incident:write, access_token:sign), heavily inspired by OAuth and NIST 8693 for token exchange in service communication.

The implementation of multi-tenancy is a critical technical aspect. Grafana chose to use OpenFGA's stores feature, assigning "each tenant their own dedicated store." This strategy provides robust data isolation, preventing "accidentally granting cross-tenant permissions," and simplifies deployment. Each OpenFGA instance is wrapped with middlewares: one for authentication to identify the user and tenant, and another to map the incoming request to the correct tenant store. Furthermore, Grafana doesn't run a single global OpenFGA instance; instead, they deploy "one OpenFGA per cluster," corresponding to each supported region. This minimizes latency and aligns with their distributed architecture.

Data migration is managed with a pragmatic, iterative approach. The concept of "feature toggles will be toggled" led to the rule of "do write always" to both the legacy and new authorization systems during the migration phase. This ensures that even if a feature is switched off and on, the state remains consistent across both systems, preventing divergence. For upgrade and downgrade cycles, particularly relevant for on-premise deployments, Grafana learned not to trust "one-time migrations." Instead, they "verify the state often between the two stores," recognizing that even a few hours of out-of-sync access control can be catastrophic. The use of shadow calls during the initial rollout is a key validation technique, where requests are sent to both the legacy and new OpenFGA engines to compare responses and ensure schema accuracy.

Finally, Grafana's deployment strategy for OpenFGA leverages their internal, open-source Grafana DSkit (Distributed Services Kit). This toolkit allows for flexible deployment: either running OpenFGA logic within the same binary, communicating in-process via gRPC over Go channels, or deploying it as a separate, independently scalable service. This flexibility caters to both simplified single-binary deployments for open-source users and highly scalable, distributed deployments for cloud environments. For their development teams, Grafana provides authlib, an internal library that simplifies interaction with the authorization system by handling caching and token exchanges, allowing developers to focus solely on defining resource permissions.

Demo / Proof of Concept

▶ Watch: Introducing OpenFGA: A modular, Zanzibar-inspired authorization system. (6:18)

While the talk did not feature a live demonstration of Grafana's migrated authorization system in action, Poovamraj Thiagarajan provided a comprehensive conceptual demonstration of how OpenFGA's declarative model works, using a "representative example of how Grafana's system works." This served as a powerful proof of concept for OpenFGA's capabilities and its direct applicability to Grafana's complex authorization needs.

The demonstration walked through the process of modeling Grafana's entities and their relationships within OpenFGA. It started by defining user and team as identities, and folder and dashboard as resources. The model illustrated how users can be associated with teams, and how folders can be nested, with dashboards residing within folders. The core of the demonstration then focused on adding authorization roles and permissions:

• Basic Assignments: A user can be an admin of a folder, or a viewer.
• Team-based Permissions: All members of a team can be automatically granted viewer access to a folder, highlighting the power of dynamic group-based access.
• Wildcard Access: Setting user:* as a viewer effectively makes a dashboard public, showcasing a simple way to manage broad access.
• Permission Definitions: An example was provided for defining a create_dashboard permission, where "only admins can create a dashboard inside a folder" is expressed directly in the model.
• Algebraic Operations: The demonstration showed how to use union operations to define viewer as anyone explicitly assigned viewer or anyone who is an admin of the folder, ensuring admins automatically inherit viewing rights.
• Inheritance: The most compelling part of the conceptual demo was illustrating how viewer permissions could inherit from parent folders to child folders, allowing for deeply nested, hierarchical access control with minimal configuration.

This detailed walkthrough effectively demonstrated OpenFGA's power, expressiveness, and declarative nature, making a strong case for its ability to handle intricate, multi-level authorization scenarios that are common in platforms like Grafana. It served as a blueprint for how Grafana is modeling its own system, proving the theoretical fit before delving into the practical migration challenges.

Defensive Implications

▶ Watch: Modeling Grafana's access control with OpenFGA examples. (7:30)

The insights from Grafana's migration to OpenFGA offer several crucial defensive implications for organizations aiming to strengthen their security posture and streamline access control.

1. Standardize and Externalize Authorization: The most significant implication is the imperative to move away from bespoke, in-application authorization logic. By adopting a standardized, externalized system like OpenFGA, organizations can eliminate authorization silos, reduce inconsistencies, and centralize policy management. This drastically reduces the surface area for authorization-related vulnerabilities, which are consistently high on the OWASP Top 10.
2. Prioritize Fine-Grained Access Control: As systems become more interconnected and distributed, coarse-grained access control is insufficient. Defenders must push for fine-grained authorization capabilities that allow precise control over who can do what to which resource, down to individual actions and objects. OpenFGA's declarative model facilitates this by allowing complex relationships and inheritance to be defined clearly.
3. Careful Schema Design is Critical for Performance and Security: While OpenFGA is powerful, the talk highlights that schema complexity can negatively impact performance. Defenders should collaborate with architects to design a schema that is expressive enough for security requirements but also optimized for performance. Overly complex schemas can lead to slow authorization checks, potentially forcing developers to bypass the system or implement insecure caching mechanisms.
4. Implement Robust Multi-tenancy Isolation: For multi-tenant platforms, using OpenFGA's stores feature per tenant is a strong defensive measure. This provides strict data isolation, preventing accidental cross-tenant data access or privilege escalation. It’s a fundamental security control that should be adopted to safeguard tenant data.
5. Adopt Phased, Verified Migration Strategies: Migrating legacy authorization systems is high-risk. Defenders should advocate for and participate in phased migration strategies that include:

• Feature toggles that write to both old and new systems, ensuring consistent behavior.
• Continuous state verification between legacy and new stores to catch synchronization issues quickly.
• Shadow calls during rollout to validate the new system's responses against the old in a production environment without impacting users.
• Incremental adoption, moving new teams and features to the new system first while maintaining stability of the old.

These techniques minimize risk and provide opportunities to detect and correct errors before they become security incidents.

1. Strategize Search with Permissions: The challenge of "search with permissions" is a common problem. Defenders need to understand the different strategies (search-and-check, filter-by-access, local indexing, aggressive pre-filtering) and their trade-offs. The choice should be based on data cardinality and performance requirements. Integrating tenant-specific filtering early in the search pipeline, as Grafana does with stack ID from ID tokens, is a powerful way to reduce the dataset and improve both security and performance.
2. Leverage Developer Tooling for Secure Adoption: Providing developers with easy-to-use libraries like Grafana's authlib is crucial. This ensures that authorization logic is consistently applied, handles caching securely, and abstracts away complexity, allowing developers to focus on correctly defining resource permissions rather than implementation details. This reduces the likelihood of developer-introduced authorization bugs.
3. Consider Deployment Flexibility: Tools like Grafana's DSkit that allow flexible deployment (in-process vs. separate service) can help organizations scale their authorization infrastructure securely. This enables independent scaling of the authorization engine, ensuring that performance bottlenecks in authorization don't impact application availability or security.

Key Takeaways

• Authorization is a pervasive, unsolved problem in the industry, leading to bespoke implementations, security gaps (OWASP Top 10), and operational overhead.
• OpenFGA, inspired by Google's Zanzibar, offers a powerful, declarative, and standardized approach to fine-grained access control, supporting complex relationships, inheritance, and algebraic operations.
• Migrating legacy, multi-tenant authorization systems to OpenFGA is challenging but feasible, requiring careful planning, iterative development, and continuous verification, as demonstrated by Grafana's 14-year legacy.
• Critical performance considerations include optimizing OpenFGA schema complexity, managing the number of tuples, and understanding the performance characteristics of underlying databases (e.g., MySQL, PostgreSQL, SQLite).
• Robust multi-tenancy is achievable by assigning dedicated OpenFGA stores per tenant, providing strong isolation and simplifying deployments by running OpenFGA instances per cluster/region.
• Effective data migration strategies involve "always write" feature toggles, continuous state verification between old and new systems, and shadow calls to validate the new authorization engine in production.

About the Speaker(s)

Jo Guerreiro is a key member of the identity and access team at Grafana. His work involves providing an application platform that enables other teams within Grafana to build features with robust authorization checks and protected resources, often instrumented with RBAC. His insights in the talk come from the practical challenges and successes of migrating Grafana's long-standing access control systems.

Poovamraj Thanganadar Thiagarajan serves as the technical lead in the FGA team. His expertise lies in integrating OpenFGA into various platforms, including CM platforms, identity platforms, and observability solutions. Poovamraj provided the foundational understanding of OpenFGA's architecture and its declarative power, setting the stage for Grafana's real-world migration story.