RContainer: A Secure Container Architecture through Extending ARM CCA Hardware Primitives

RContainer: A Secure Container Architecture through Extending ARM CCA Hardware Primitives

Speakers: Qihang Zhou, Institute of Information Engineering Chinese Academy of Science

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=nYa-RrHYBK8

Overview

The proliferation of containers in modern cloud computing environments has brought significant benefits in terms of efficient deployment and high resource utilization. However, their inherent weak isolation mechanisms have consistently posed substantial security challenges. This talk, "RContainer: A Secure Container Architecture through Extending ARM CCA Hardware Primitives," presented by Qihang Zhou, addresses these critical security concerns by proposing a novel container architecture that leverages the advanced hardware primitives of ARM Confidential Compute Architecture (CCA). The core problem RContainer seeks to solve is achieving strong isolation for containers, even against a compromised host OS, while simultaneously minimizing the Trusted Computing Base (TCB) and maintaining low performance overhead.

Traditional container solutions often struggle with the delicate balance between robust security and practical performance. Existing approaches either compromise isolation by sharing a single kernel or introduce excessive overhead and TCB by assigning a dedicated OS to each container. RContainer introduces a mini OS running in EL1 alongside a de-privileged host OS, coupled with a unique shim-style isolation mechanism. This design aims to limit the impact of containers on the kernel, significantly reducing the attack surface and providing enhanced security guarantees.

This work is particularly relevant in an era where cloud security is paramount and hardware-assisted security features are becoming increasingly sophisticated. By extending ARM CCA's granular protection mechanisms, RContainer offers a compelling vision for future containerization platforms that demand both high performance and ironclad isolation. It represents a significant step towards enabling confidential computing for containerized workloads, making it a critical development for cloud providers, enterprises, and anyone deploying sensitive applications in multi-tenant environments.

Background

▶ Watch: Introduction: Container security and weak isolation problem (0:00)

The security landscape for containers has long been characterized by a fundamental tension: the efficiency and resource utilization benefits of containers are often at odds with their inherent isolation weaknesses. Unlike virtual machines, containers typically share the host operating system's kernel, making them susceptible to various attacks if the host OS is compromised. This "weak isolation" means that a vulnerability in the kernel or a malicious co-resident container could potentially affect all other containers on the same host. The challenge is not merely to protect containers from a compromised host OS, but to achieve strong isolation with a minimal TCB and acceptable performance overhead.

Prior attempts to enhance container security have explored various avenues. Some approaches involve running containers within lightweight virtual machines (e.g., Kata Containers, gVisor), which increases isolation but often introduces significant performance overhead and a larger TCB. Others have focused on kernel hardening or specialized sandboxing techniques, which can mitigate some risks but may not offer the hardware-backed guarantees required for true confidential computing.

The advent of ARM v9-A and its Confidential Compute Architecture (CCA) introduces a new paradigm for hardware-enforced isolation. ARM CCA defines four distinct physical address spaces: the Normal World, Secure World, and the newly added Realm World. A key component of ARM CCA is the Granular Protection Table (GPT), managed by firmware in EL3 (the highest privilege level). The GPT provides fine-grained memory protection by defining access permissions for physical address spaces, acting as a secondary access check after the Memory Management Unit (MMU). If the GPT denies access, it overrides the MMU, making CCA isolation both flexible and powerful.

However, directly applying ARM CCA to containers presents its own set of challenges. Running multiple containers within a single Realm OS would mean they share the same kernel, undermining the strong isolation CCA aims to provide. Conversely, assigning each container its own Realm OS would be prohibitively heavy, introducing substantial overhead and TCB, making it impractical for large-scale deployments.

The placement of the TCB itself is another critical design decision. Deploying the TCB in the Secure or Realm World can lead to frequent world-switch overheads and a large TCB. Alternatively, placing the TCB in the Normal World, controlled by page tables (as seen in nested kernel approaches), can result in frequent Translation Lookaside Buffer (TLB) flushing overhead. Both options are suboptimal.

RContainer's threat model assumes an initially safe system that may later be compromised. This includes potential compromises of the container OS, hypervisor, and other system components. While physical side-channel and Denial-of-Service (DoS) attacks are acknowledged as outside the scope of RContainer's direct mitigation, the primary focus is on protecting containerized workloads from a compromised host OS and ensuring strong isolation between co-resident containers.

Key Findings

▶ Watch: Challenges of using ARM CCA for containers (2:30)

RContainer introduces a novel secure container architecture that directly addresses the limitations of existing solutions by leveraging and extending ARM CCA hardware primitives. The core innovation lies in its ability to protect containers on an untrusted host OS and enforce strong isolation within a minimal TCB, all while maintaining low performance overhead and requiring no modifications to container images.

The key findings and contributions of RContainer can be summarized as follows:

1. Strong Isolation with Minimal TCB: RContainer achieves robust isolation between containers in both user space and kernel space by employing a mini OS in EL1 and a shim-style isolation mechanism. This design ensures that even if the host OS is compromised, the containers remain protected. The TCB, particularly in the highest privilege level (EL3), is remarkably small, consisting of only 130 lines of code (LoC). The total TCB across all privilege levels is 2,600 LoC, significantly smaller than comparable secure execution environments like Shelter, which has over 2,000 LoC in EL3 alone and continues to grow.

1. Efficient Resource Utilization and Low Overhead: Despite providing strong security guarantees, RContainer demonstrates impressive performance. Application workloads evaluated on a hardware prototype showed less than 10% overhead compared to unprotected execution. This performance is notably superior to traditional virtualization methods for container isolation. Specifically, RContainer reduced average overhead by 5.7% compared to the Shelter project, highlighting its efficiency.

1. Leveraging ARM CCA for Granular Protection: RContainer effectively utilizes ARM CCA's Granular Protection Table (GPT) to enforce memory isolation. By assigning different GPTs to the mini OS, the de-privileged host OS, and individual container components (referred to as "consumes"), RContainer creates distinct, hardware-enforced memory regions. This mechanism ensures that each entity can only access its designated memory, preventing unauthorized access and data leakage.

1. Prototype Implementation and Evaluation: The project developed two prototypes: an FVP prototype for security evaluation and a hardware prototype (based on the RK3399 ARM V8 development board) for performance evaluation. Security analysis, including testing against approximately 30 CVEs, revealed that most attacks occur at runtime. RContainer's design concentrates its minimal TCB on runtime components, effectively mitigating this attack vector. The analysis showed that only about 2.7% of the ARM Trusted Firmware (ATF) code is dedicated to runtime, demonstrating the efficacy of minimizing the runtime TCB.

In essence, RContainer presents a practical and high-performance solution for securing containerized workloads in confidential computing environments, offering a compelling alternative to existing methods by strategically combining hardware-assisted isolation with a carefully minimized software TCB.

Technical Deep Dive

▶ Watch: Mini OS protection via mixed page tables and GPTs (5:30)

RContainer's architecture is meticulously designed to provide strong isolation and minimize the TCB by extending ARM CCA hardware primitives. It introduces two primary innovations: a mini OS running in EL1 and a shim-style isolation mechanism.

ARM CCA Primitives and RContainer's Approach

ARM CCA, introduced in ARM v9-A, is fundamental to RContainer. It defines multiple privilege levels (EL0-EL3) and address spaces. EL3 is the highest privilege level, typically running a secure monitor or firmware. EL1 is where operating systems typically run, and EL0 is for user applications. The Granular Protection Table (GPT) is a critical hardware feature, managed by EL3 firmware, that provides an additional layer of memory access control beyond the traditional Memory Management Unit (MMU). The GPT allows defining precise access permissions for physical memory regions, crucial for enforcing strong isolation.

RContainer deviates from the conventional use of Realm World for containers, which would either lead to weak isolation (shared kernel) or excessive overhead (OS per container). Instead, it deploys a compact mini OS in EL1 alongside a de-privileged, untrusted host OS, also running in EL1. The mini OS acts as a security enforcement layer, controlling access to resources and mediating interactions.

The Mini OS

The mini OS is a compact, basic operating system running in EL1. Its primary responsibilities include:

1. Mixed Page Table for Tamper-Proof Protection: RContainer utilizes the same MMU page table for all components (mini OS, host OS, containers). However, it assigns them different GPTs (or different branches within the LZO page table).

• The privileged GPT is used by the mini OS.
• The OS GPT is used by the traditional, de-privileged host OS.
• Crucially, the mini OS-related memory access is explicitly set to "no access" in the OS GPT. This means the de-privileged host OS cannot access the mini OS's memory, ensuring its integrity and confidentiality.
• This "mixed page table" approach provides hardware-enforced memory isolation for the mini OS itself, protecting it from a potentially compromised host OS.

1. Lightweight Memory Allocator: The mini OS maintains the GPT at a software level and includes a lightweight memory allocator. This custom allocator is optimized for the specific needs of RContainer's security architecture, managing memory within its secure domain.

1. Control Flow Protection: The mini OS incorporates a control flow protector, responsible for:

• Exception Interposing: Intercepting exceptions and system calls to enforce security policies.
• Context Switching: Securely switching between the de-privileged host OS and different container contexts, ensuring proper isolation during transitions.

Shim-Style Isolation

RContainer's shim-style isolation focuses on strengthening security within the kernel data plane, based on the observation that while most attacks originate in the control plane, the data plane requires stronger isolation for containers.

The key concept here is the introduction of "consumes." The speaker clarifies that a "consum" represents an "important procedure out of the de-privileged OS" or "an instance of some function of the host OS." These consumes are instantiated within the kernel data plane and are critical components closely tied to a container's execution.

1. Per-Consum GPT: Each consum (and its associated container memory) is assigned a separate GPT. This is a powerful mechanism:

• A consum is strictly limited to accessing only its own memory.
• It cannot access the memory of other containers' consumes.
• It cannot access the mini OS's memory.
• It cannot access the de-privileged host OS's memory.
• This provides robust, hardware-enforced memory isolation at a very granular level, effectively preventing cross-container or container-to-host memory attacks. The container and its consums are treated as a protected combination.

Container Life Cycle Protection

RContainer enforces security throughout the container's life cycle:

1. Boot Process:

• The de-privileged host OS is initially loaded and measured by the EL3 secure monitor, establishing a trusted boot chain.
• The system then boots normally.
• The mini OS takes control to allocate memory for the consumes associated with containers.
• It records critical information such as the system call stack, shared memory regions, and private data for each container.

1. Task Creation: When a new task (e.g., a new process within a container) is created, the mini OS:

• Validates the address of the new task structure and its associated page table.
• Locks these structures, preventing unauthorized modification.

1. Task Termination: Upon task termination, the mini OS:

• Deletes the task structure.
• Clears all associated memory, preventing residual data exposure.

Memory Page Fault Handling

To ensure smooth operation and minimize performance impact, RContainer implements a two-tiered memory page fault handling workflow:

1. Fast Page Fault Workflow:

• The mini OS maintains a secure memory pool within its protected domain.
• When a page fault occurs, a lightweight pageable handler within the mini OS attempts to allocate memory from this secure pool. This is the primary and most efficient path.

1. Slow Page Fault Workflow:

• If the secure memory pool in the mini OS is exhausted, the workflow falls back to the de-privileged host OS.
• The traditional memory management handler of the host OS is then used to allocate memory. This separation ensures that even when relying on the untrusted OS for memory, the overall security model remains intact due to the strict GPT-based isolation.

IO Protection

RContainer's current scope for IO protection is specific:

• DMA Attack Protection: RContainer can protect containers against Direct Memory Access (DMA) attacks. This is achieved by extending the System Memory Management Unit (SMMU) with GPT capabilities, allowing the SMMU GPT to limit access to SMMU memory regions, thus preventing malicious DMA operations from compromising container memory.
• Disk and Network IO: Disk IO and network IO are explicitly stated as not being within RContainer's direct scope for protection at the container level. The rationale is that application-layer protection is preferred for these types of IO data, suggesting that RContainer focuses on lower-level memory and control flow integrity.

This detailed architecture demonstrates RContainer's comprehensive approach to securing containers by integrating hardware-backed isolation with a carefully designed software layer.

Demo / Proof of Concept

▶ Watch: Shim-style isolation for container kernel data plane (7:00)

The RContainer project developed two distinct prototypes to validate its security claims and evaluate its performance, demonstrating the viability of its architecture.

FVP Prototype for Security Evaluation

A Fixed Virtual Platform (FVP) prototype was developed based on ARM FVP. This virtualized environment allowed for rigorous security evaluation without the complexities of physical hardware. The team conducted extensive testing against approximately 30 Common Vulnerabilities and Exposures (CVEs). The analysis of these CVEs yielded a crucial insight: the majority of attacks occur at runtime. This finding underscored the importance of minimizing the TCB specifically for runtime components.

The security evaluation highlighted RContainer's success in reducing the attack surface. The team found that only about 2.7% of the ARM Trusted Firmware (ATF) code, which runs at higher privilege levels, is actually critical for runtime operations. The remaining code, such as that involved in the boot process, is covered after loading and thus less susceptible to runtime attacks once the system is initialized. This analysis supported the design principle of keeping the runtime TCB extremely compact.

Hardware Prototype for Performance Evaluation

To assess real-world performance overhead, a hardware prototype was implemented on a Soft Hardware Development Board named RK3399. This board utilizes an ARM V8 architecture, providing a realistic platform for measuring RContainer's efficiency.

Performance evaluations were conducted using a suite of standard application workloads. The results were highly encouraging: RContainer introduced less than 10% overhead on these real-world applications. This figure is significantly better than the overhead typically associated with full virtualization methods used for container isolation.

Furthermore, RContainer's performance was directly compared against Shelter, another prominent secure execution environment. The evaluation showed that RContainer achieved an average overhead reduction of 5.7% compared to Shelter, indicating its superior efficiency in practical scenarios.

TCB Size Comparison

A critical aspect of RContainer's proof of concept is its minimal TCB. The implementation results showed:

• Total TCB: Approximately 2,600 lines of code (LoC) across all privilege levels.
• EL3 TCB: Only 130 LoC in the highest privilege level (EL3). This is a remarkable achievement, as EL3 is the most critical component for system security.
• Comparison with Shelter: In contrast, Shelter, a comparable project, has over 2,000 LoC in EL3, and its TCB is noted to be continuously growing as new security features are added. RContainer's ability to maintain such a small EL3 TCB significantly reduces the attack surface and simplifies security auditing.

The dual prototype approach, combining comprehensive security testing in a virtual environment with detailed performance measurement on real hardware, provides strong empirical evidence for RContainer's effectiveness and practicality.

Defensive Implications

▶ Watch: RContainer implementation and evaluation prototypes (10:00)

RContainer presents a significant advancement for defenders seeking to enhance the security of containerized workloads, particularly in cloud and multi-tenant environments where trust boundaries are critical. The core defensive implications revolve around leveraging hardware-backed isolation to mitigate risks from a compromised host OS and enforce strong inter-container separation.

1. Hardware-Backed Isolation Against Host Compromise: The most profound implication is the ability to protect containers even when the underlying host OS is compromised. By running a mini OS in EL1 with its memory protected by a dedicated GPT (inaccessible to the de-privileged host OS), RContainer establishes a robust hardware-enforced barrier. This is a game-changer for cloud providers and enterprises deploying sensitive applications, as it substantially reduces the attack surface presented by the host kernel. Defenders can now achieve a higher level of assurance that their containerized applications remain confidential and integral, even if the host environment is breached.

1. Granular Inter-Container Isolation: RContainer's shim-style isolation with per-consum GPTs provides unprecedented granularity in separating containers. Each container, along with its critical kernel-level components ("consumes"), is confined to its own memory space, preventing lateral movement and data leakage between co-resident containers. This significantly mitigates the risk of one compromised container affecting others on the same host, a common vulnerability in traditional container deployments. Defenders can deploy multi-tenant applications with greater confidence in their isolation properties.

1. Reduced Trusted Computing Base (TCB): The minimal TCB, particularly the 130 LoC in EL3, is a critical defensive advantage. A smaller TCB means fewer lines of code to audit, fewer potential vulnerabilities, and a higher likelihood of proving correctness. This simplifies security assurance efforts and reduces the overall risk profile of the system. Defenders should prioritize solutions that minimize TCB, and RContainer's design aligns perfectly with this principle.

1. Protection Against DMA Attacks: While RContainer's scope for disk and network IO is limited, its ability to protect against DMA attacks via SMMU GPT is a vital defensive capability. DMA attacks can bypass traditional CPU-centric security mechanisms, making them a potent threat. By extending GPT protection to the SMMU, RContainer helps secure container memory from malicious peripherals or compromised device drivers.

1. Low Performance Overhead for Practical Deployment: The demonstrated less than 10% overhead is crucial for adoption. High security often comes with a performance penalty, making it impractical for many production environments. RContainer's efficiency means that organizations do not have to choose between strong security and acceptable performance, enabling wider deployment of confidential computing principles for containerized workloads.

1. Future-Proofing with ARM CCA: By building directly on ARM CCA, RContainer aligns with the future direction of ARM-based confidential computing. Defenders investing in ARM architectures can leverage this work to build more secure platforms that benefit from ongoing hardware security enhancements.

In summary, RContainer offers a powerful toolkit for defenders, enabling them to deploy containerized applications with significantly enhanced security guarantees, a reduced attack surface, and acceptable performance, making it an attractive solution for the evolving threat landscape in cloud environments.

Key Takeaways

• Strong Isolation via Hardware Primitives: RContainer achieves robust isolation for containers, even against a compromised host OS, by extending ARM CCA's Granular Protection Table (GPT) to create hardware-enforced memory boundaries.
• Minimal Trusted Computing Base (TCB): The architecture boasts an exceptionally small TCB, with only 130 lines of code (LoC) in the highest privilege level (EL3) and a total of 2,600 LoC, significantly reducing the attack surface compared to other secure execution environments.
• Efficient Performance: RContainer demonstrates low performance overhead, with less than 10% impact on application workloads and an average 5.7% reduction in overhead compared to the Shelter project, making it practical for production use.
• Innovative Mini OS and Shim-Style Isolation: A compact mini OS runs in EL1, protected by its own GPT, while "shim-style isolation" uses per-container "consumes" with dedicated GPTs to ensure granular memory separation between co-resident containers.
• Comprehensive Life Cycle Protection: Security is enforced throughout the container's life cycle, from boot-time measurement by EL3 to secure task creation and termination, ensuring integrity and confidentiality at every stage.
• DMA Attack Mitigation: RContainer provides protection against Direct Memory Access (DMA) attacks by extending SMMU GPT capabilities, securing container memory from malicious peripherals or compromised drivers.

About the Speaker(s)

Qihang Zhou is affiliated with the Institute of Information Engineering, Chinese Academy of Science. Their research focuses on enhancing the security of modern computing environments, particularly in the realm of containerization and confidential computing. This work on RContainer exemplifies their expertise in leveraging advanced hardware security features, such as ARM CCA, to address critical challenges in cloud security and system isolation.