Private Hierarchical Governance for Encrypted Messaging

Private Hierarchical Governance for Encrypted Messaging

Speakers: Armin Namavari (Cornell Tech, University of Washington); Barry Wang; Sanketh Menda; Ben Nassi; Nirvan Tyagi; James Grimmelmann

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=eqeuzWHZi94

Overview

End-to-end encrypted (E2EE) messaging has become a cornerstone of digital privacy, deployed to billions of users across platforms like WhatsApp. This technology ensures that message content remains confidential, accessible only to the communicating endpoints, even in the event of a service provider compromise by hackers or malicious insiders. While this represents a significant victory for user privacy against external threats, it introduces a complex challenge for mitigating online abuse, such as misinformation, spam, and harassment, which does not require server compromise. The very strength of E2EE—its opacity to the service provider—makes traditional content-based moderation impossible.

The talk "Private Hierarchical Governance for Encrypted Messaging," presented by Armin Namavari at IEEE S&P, introduces a novel framework designed to address this critical gap. The proposed system, Private Hierarchical Governance, empowers communities within E2EE platforms to define and enforce their own norms of acceptable behavior through automated policies and community-specific moderation tools. Crucially, this is achieved while preserving the core privacy guarantees of end-to-end encryption, ensuring that governance-related state remains private from the platform until explicitly escalated.

This work not only expands the solution space for abuse mitigation in E2EE environments but also navigates the nuanced tension between user privacy and online safety. By shifting governance logic to client devices and building upon existing E2EE protocols like Messaging Layer Security (MLS), the framework offers a pragmatic and scalable approach. It addresses the diverse moderation needs of various communities, from dorm group chats to activist organizing groups, allowing for context-specific policies that are currently lacking in platform-centric, "one-size-fits-all" moderation models. The research highlights a promising new direction for enhancing user safety and empowering communities within the secure confines of end-to-end encryption.

Background

▶ Watch: Introduction: The problem of abuse in E2EE messaging (0:00)

The widespread adoption of end-to-end encryption has brought immense privacy benefits, yet it has simultaneously amplified the challenge of combating online abuse. A recent paper by Thomas et al. (referenced in the talk) found that 48% of surveyed users had experienced some form of online harassment, with a 1.3 times increase in the odds of experiencing harassment over time, underscoring the growing severity of this problem. E2EE platforms are not immune; examples like the spread of viral misinformation on WhatsApp demonstrate the need for effective mitigation strategies.

Prior attempts at abuse mitigation in E2EE contexts have encountered significant limitations and controversy. One approach involves analyzing message traffic and metadata to predict abusive behavior, as seen with WhatsApp's spam detection based on message frequency. However, this method is inherently limited, as metadata often provides an insufficient signal for detecting nuanced abuse categories like misinformation, and it completely rules out content-based moderation.

Another set of proposals, notably for privacy-preserving client-side scanning (e.g., Apple's controversial plans for detecting Child Sexual Abuse Material or CSAM), aimed to scan messages against known harmful content lists on the user's device, with automatic reporting upon detection. These proposals were met with strong backlash from academia and civil society, raising concerns about potential backdoors into E2EE and a lack of transparency regarding the content blocklists. The fear was that such mechanisms could be expanded to censor other forms of content, undermining the fundamental privacy promise of E2EE.

User-driven content reporting, where users manually report harmful content to the platform, is another existing mechanism. While more respectful of user agency, it is inherently reactive, occurring only after the harmful content has been received and viewed, and the damage potentially inflicted. Furthermore, platform-centered reporting often provides a "one-sized fits-all" approach to moderation that fails to account for the context-specific considerations of diverse communities. This is particularly evident in the lower quality of moderation observed in languages other than English.

The limitations of these existing approaches highlight a critical need for a new paradigm. The success of community-driven moderation on plaintext platforms like Reddit and Discord, where community moderators and automated policies play a central role, suggests a viable alternative. These platforms benefit from moderators who possess intimate knowledge of their community's context and content. The challenge, then, is to transpose this model of decentralized, context-aware governance into the E2EE realm, where the platform itself cannot observe content or moderation policies, thereby preserving user privacy while enabling effective abuse mitigation.

Key Findings

▶ Watch: Proposing Private Hierarchical Governance for abuse mitigation (3:59)

The core contribution of this research is the introduction of Private Hierarchical Governance, a novel framework for abuse mitigation in end-to-end encrypted messaging that addresses the shortcomings of prior approaches. This framework is characterized by several key findings and design principles:

1. Community-Centric Policy Enforcement: The system empowers communities to define and enforce their own "norms of acceptable behavior" through automated policies. This allows for highly contextualized moderation, where a dorm group chat and an activist organizing chat can benefit from vastly different moderation tools, structures, and policies. Examples include community-specific word filters, special privileges for moderators, or voting mechanisms for new moderators.
2. Privacy-Preserving Governance State: A central tenet is that all governance-related state—such as the identity of moderators, the contents of a word filter, or within-community reporting—remains private from the platform. This state is only escalated to the platform if community members decide to do so, aligning perfectly with the privacy goals of end-to-end encryption.
3. Hierarchical Moderation: The framework supports a hierarchical model where community moderators handle first-line moderation, possessing content and context that the platform does not. If a community moderator deems content to violate broader platform guidelines, they can escalate it to a platform moderator, who can then take platform-level actions (e.g., temporary bans).
4. Three High-Level Security and Privacy Goals: The design targets:

• Accountability: The reporting mechanism disallows report forgery, and all sent messages are digitally signed and reportable.
• Governance Privacy: Governance-related state and within-community reporting are kept private from the platform.
• Governance Integrity: All clients share a consistent view of common governance state (e.g., moderator identities, word filter contents), even in the presence of malicious clients.

1. Architectural Shift to Client Devices: To realize community-driven governance in an E2EE setting, the system proposes a fundamental architectural shift, moving governance logic and state directly to client devices. This departs from the common platform-hosted moderation infrastructure prevalent in plaintext settings.
2. Extensible and Scalable Design: The approach is designed to be extensible, allowing for varied governance possibilities across diverse communities without forcing a single model. It also aims to be performant and scale to large group sizes, essential for mass-deployed E2EE applications.
3. Leveraging MLS for Ordered State: The framework leverages and extends Messaging Layer Security (MLS), a recently standardized E2EE messaging protocol. MLS provides a robust commit mechanism for consistent updates of shared cryptographic state (like group membership and keys). The innovation lies in extending MLS to support ordered governance messages, enabling consistent updates to shared, encrypted governance state.

In essence, Private Hierarchical Governance demonstrates that it is possible to introduce sophisticated, community-driven moderation capabilities into E2EE environments without compromising the fundamental privacy principles that E2EE is built upon.

Technical Deep Dive

▶ Watch: Example scenario of community moderation in action (4:40)

The technical realization of Private Hierarchical Governance centers on a novel architectural approach that shifts governance logic and state to client devices, mediating between the application layer and the underlying end-to-end encryption protocol. This is a significant departure from traditional centralized moderation.

The system is designed with a layered view of applications. At its core is a governance layer that sits between application-level actions (e.g., a user sending a message, a moderator applying a filter) and the E2EE messaging protocol. This governance layer is responsible for maintaining an encrypted state that is synchronized among clients and updated through special messages that adhere to a consistent ordering property.

The choice of the underlying E2EE protocol is critical. The researchers opted for and extended Messaging Layer Security (MLS), an IETF standard for asynchronous E2EE messaging. MLS is well-suited because it provides robust group management, encryption of messages with forward secrecy (preventing past messages from being decrypted if a key is compromised) and post-compromise recovery (allowing a group to recover security after a member's key is compromised). While standard MLS does not guarantee the ordering of messages containing user content (for performance and asynchronous network compatibility), it does include a proposal and commit mechanism for consistent updates of shared cryptographic state related to group membership. All group members must observe and process the same ordering of these commit messages, which can update shared keys and change the list of group members. This provides a crucial consensus mechanism that the Private Hierarchical Governance framework builds upon.

The key technical extension to MLS involves augmenting its API to allow for the separation of ordered and unordered application messages.

• Ordered messages are specifically used for updating the shared governance state. These messages leverage the existing MLS commit mechanism, ensuring that all honest clients process them in the same sequence, thereby maintaining governance integrity.
• Unordered messages continue to carry user content within the group, without strict ordering guarantees, preserving MLS's performance benefits for general communication.

The governance layer manages a consistent shared state among clients. This state is end-to-end encrypted and contains critical information for community moderation. It includes Role-Based Access Control (RBAC) information, which keeps track of user privileges within a community (e.g., who is a regular member, who is a moderator, who can ban users, who can modify policies). This RBAC generalizes prior work on group permissions for encrypted messaging. In addition to RBAC, the shared state also holds policy-specific information, such as the contents of a word filter or a table of user reputations (though with a crucial caveat discussed below). By transmitting these state updates through encrypted commit messages, the system guarantees both the confidentiality and consistency of the governance state.

To facilitate reporting, all messages, both ordered and unordered, have a digital signature attached. This signature is sent in the event of a report, ensuring accountability by disallowing report forgery and making all sent messages verifiably attributable.

A notable limitation imposed by the design's efficiency requirements is that policies cannot safely update their state as a function of unordered messages (i.e., user content). Because user content messages lack strong consistency guarantees, a policy cannot reliably use them to update shared governance state across all clients. This means, for instance, that a reputation system that is a function of content posted within the group is not feasible within this design. However, a reputation system that is a function only of governance messages (e.g., reports, moderator actions, votes) would be feasible. Despite this constraint, the design still enables a wide range of useful policies, including mutable word filters, voting for new moderators, and instituting special permissions.

The governance layer provides a structured policies-as-code paradigm inspired by the PolicyKit system from plaintext settings. This framework allows developers to define policies that specify:

1. What group actions are in scope for those policies (e.g., sending a message, attempting to change a filter).
2. How those actions are accepted, blocked, or stored for later approval (as in the case of voting).

The security properties of the system are robust:

• Governance Integrity is achieved through the MLS transcript hash mechanism, which ensures clients agree on the ordering of commits, combined with consistency checks performed by newly invited group members. Malicious clients cannot corrupt the view of honest clients regarding the shared governance state.
• Governance Privacy is guaranteed by MLS's strong confidentiality properties, ensuring that unreported messages and governance information remain private from a malicious platform.
• Accountability for reports is provided by the security of the digital signature scheme used, preventing report forgery.

The authors acknowledge certain areas not yet covered: traffic analysis implications are not considered, as addressing them would likely require more powerful techniques like metadata-private communication. Deniability is also not a current goal, though reporting tools like asymmetric message ranking could be adapted. Formal analysis of the entire system is also identified as future work, building on ongoing efforts to formally verify the MLS protocol itself.

Demo / Proof of Concept

▶ Watch: Benefits of community-driven moderation and its importance (6:30)

To demonstrate the practical feasibility and performance of their Private Hierarchical Governance framework, the researchers implemented a proof of concept system. This implementation was built in Rust on top of the open MLS library, which serves as a concrete realization of the MLS standard.

The changes required to the underlying MLS protocol were described as "small," primarily amounting to the introduction of a new proposal type for ordered application messages. This minimal modification underscores that the core architectural innovation can be integrated efficiently into existing MLS implementations without necessitating a complete overhaul of the protocol.

The governance layer itself, which encapsulates the logic for managing shared state, RBAC, and policy enforcement, comprised approximately 4,000 lines of code. This figure provides a tangible measure of the complexity and scope of the governance framework built on top of the extended MLS.

Performance evaluation was a critical component of the proof of concept. The researchers found that adding governance capabilities incurred only a small overhead in terms of latency and bandwidth. This is a crucial finding, as performance degradation would be unacceptable for widely deployed E2EE messaging applications. Even for more involved policies, such as voting mechanisms, the system demonstrated practical performance.

As a specific example, the talk highlighted a scenario involving a group of 1,024 members simultaneously casting votes. In this large-scale test, a vote was observed to complete in roughly two seconds. This impressive performance was attributed to a special batching optimization employed within the system, which efficiently processes multiple governance updates. This result strongly indicates that the Private Hierarchical Governance approach is practical and scalable for large groups with high message traffic, aligning with the core aims of massively deployed E2EE encryption.

The successful implementation and performance evaluation of the proof of concept solidify the claim that Private Hierarchical Governance is not merely a theoretical construct but a viable and efficient solution for enhancing abuse mitigation in end-to-end encrypted messaging.

Defensive Implications

▶ Watch: Architectural shift: Client-side governance logic (8:00)

The Private Hierarchical Governance framework offers profound defensive implications for both users and platform providers in the E2EE ecosystem. Its primary strength lies in empowering communities to become the first line of defense against online abuse, without sacrificing the privacy guarantees of end-to-end encryption.

For users and communities, this framework provides unprecedented agency and control. Instead of being subject to a generic, often inadequate, platform-wide moderation policy, communities can:

• Tailor Policies to Context: A dorm chat, an activist group, or a family chat can each define moderation policies (e.g., specific word filters, rules for sharing information, acceptable discourse) that are perfectly suited to their unique context, culture, and risk profile. This addresses the "one-size-fits-all" problem inherent in platform-centric moderation.
• Proactive Abuse Mitigation: Communities can implement proactive measures like mutable word filters, preventing harmful language from ever being seen. This shifts the paradigm from purely reactive reporting (where damage has often already occurred) to a more preventative stance.
• Strengthen Internal Trust: By having transparent, community-defined rules and a clear process for internal moderation, groups can foster a stronger sense of trust and safety among their members.
• Protect Sensitive Governance Information: The privacy of governance state is paramount. For groups like activist organizations, the identity of moderators or the specific policies they enact can be highly sensitive. Keeping this information private from the platform itself protects these groups from potential targeting by malicious actors or corrupt governments.

For platform providers, Private Hierarchical Governance offers several strategic advantages:

• Reduced Moderation Burden: By offloading first-line moderation responsibilities to communities, platforms can significantly reduce the immense operational and psychological burden on their own moderation teams. This allows platform moderators to focus on severe, escalated cases that violate broader legal or platform guidelines.
• Improved User Satisfaction: Empowered communities are likely to be more satisfied with their messaging experience, as their specific needs for safety and discourse are met more effectively. This can lead to increased engagement and loyalty.
• Ethical Compliance and Backlash Mitigation: By offering a privacy-preserving abuse mitigation solution, platforms can avoid the ethical dilemmas and public backlash associated with client-side scanning proposals that risk undermining E2EE. This framework demonstrates a path to online safety that respects fundamental privacy rights.
• Scalability for Diverse Needs: As E2EE becomes more pervasive across various platforms and use cases (e.g., federated systems, interoperable platforms), this framework provides a scalable model to integrate abuse mitigation without having to re-engineer centralized moderation for every new context.

In essence, defenders should consider adopting the principles of Private Hierarchical Governance for any E2EE application. It represents a paradigm shift from centralized, opaque moderation to decentralized, transparent, and community-driven safety. By integrating this framework, E2EE platforms can offer a more secure and trustworthy environment, where users are protected not just from external threats, but also from internal abuse, all while maintaining the integrity of their privacy guarantees.

Key Takeaways

• Empowering Community-Driven Moderation: Private Hierarchical Governance enables communities within end-to-end encrypted (E2EE) platforms to define and enforce their own context-specific moderation policies, moving beyond "one-size-fits-all" platform rules.
• Preserving E2EE Privacy: Crucially, all community-specific governance state (e.g., moderator identities, word filters, internal reports) remains end-to-end encrypted and private from the platform, aligning with core E2EE principles, until explicitly escalated by community members.
• Leveraging MLS for Consistent State: The framework extends the Messaging Layer Security (MLS) protocol to support ordered governance messages, utilizing MLS's commit mechanism to ensure all clients maintain a consistent and confidential view of the shared governance state.
• Separation of Ordered and Unordered Messages: The system introduces a novel separation between ordered messages (for governance state updates) and unordered messages (for user content), balancing the need for consistency in governance with the performance requirements of asynchronous E2EE communication.
• Accountability and Integrity: Digital signatures on all messages prevent report forgery, ensuring accountability. MLS's robust properties guarantee governance integrity, meaning malicious clients cannot corrupt the shared governance view of honest clients.
• Proven Feasibility and Scalability: A proof of concept implemented in Rust on open MLS demonstrated minimal performance overhead. For example, a voting mechanism in a group of 1,024 members completed in approximately two seconds due to batching optimizations, proving its practicality for large, active groups.

About the Speaker(s)

The primary presenter for this talk was Armin Namavari, who introduced himself as a researcher collaborating with teams at Cornell Tech and the University of Washington. His work, as presented, focuses on building innovative solutions for governance within encrypted messaging, particularly in the context of abuse mitigation.

Armin Namavari was joined in this joint work by several collaborators: Barry Wang, Sanketh Menda, Ben Nassi, Nirvan Tyagi, and James Grimmelmann. While their specific affiliations were not individually detailed in the transcript, the collective effort was attributed to researchers at Cornell Tech and the University of Washington, indicating a collaborative academic research project. Their combined expertise contributed to the development and analysis of the Private Hierarchical Governance framework, aiming to advance the state of online safety while preserving user privacy in end-to-end encrypted communication systems.