Bug Hunting In VMware Device Virtualization

Bug Hunting In VMware Device Virtualization

Speakers: JiaQing Huang, Security Research, Tiangong team at Qianxin Group; Hao Zheng, Security Research, Tiangong team at Qianxin Group; Yue Liu, Team Leader, Tiangong team at Qianxin Group

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=sfma7ymwmdA

Overview

This talk, "Bug Hunting In VMware Device Virtualization," delivered by JiaQing Huang and Hao Zheng from the Tiangong team at Qianxin Group, offers a comprehensive guide for security researchers looking to delve into VMware virtualization security. The speakers, alongside their team leader Yue Liu, share their journey and methodologies, emphasizing a holistic approach to reverse engineering the entire VMware virtualization architecture rather than solely focusing on individual virtual device code. This perspective is crucial for identifying how guest operating systems can influence and exploit underlying host components.

The presentation provides invaluable insights into the intricacies of VMware Workstation and ESXi, detailing critical components like the VMX process, User RPC, and Shared Area mechanisms. It outlines a structured methodology for bug hunting, leveraging their experience which includes successfully escaping Parallel Desktop at GeekCon 2023 and reporting numerous bugs to VMware. The talk serves as both a practical guide and an inspiration for newcomers, demonstrating that despite the complexity, virtualization security research is a challenging yet rewarding field with significant impact on system integrity.

The core message underscores that virtual devices remain a highly effective attack surface for discovering critical vulnerabilities. By dissecting the architectural interaction points between the virtual machine monitor (VMM) and the host-side VMX process, the speakers illustrate how a deeper understanding of these foundational mechanisms can unveil new avenues for exploitation. This detailed exploration is particularly relevant in an era where virtualization technology underpins cloud infrastructure and enterprise environments, making its security paramount.

Background

▶ Watch: Introduction, team, and motivation for virtualization security research (0:00)

Virtualization technology forms the backbone of modern computing, from personal workstations to massive cloud data centers. Its pervasive nature also makes it a prime target for sophisticated adversaries and a frequent challenge in hacking contests, where successful escapes from virtualized environments represent significant achievements. Despite its critical role, the complexity of virtualization introduces a vast attack surface, particularly within the myriad of emulated devices that bridge the guest and host environments.

The speakers embarked on their virtualization security research journey at the end of 2022, initially with limited prior knowledge. Their motivation stemmed from the inherent challenge and the desire to understand the underlying mechanisms through reverse engineering. This intensive period of research quickly yielded substantial results, including a successful escape from Parallel Desktop at GeekCon 2023 and the responsible disclosure of numerous bugs affecting both VMware Workstation and ESXi to VMware. These early successes underscored the potential for significant findings within this domain.

Traditionally, bug hunting in virtualization often focuses on debugging the code for specific virtual devices. However, the Tiangong team adopted a broader, more architectural approach. They recognized that a deeper understanding of the entire VMware virtualization architecture – how the hypervisor interacts with the host OS, how guest requests are processed, and how data is shared – would reveal more profound vulnerabilities. This strategy aimed to identify how the guest OS could influence host-side code beyond simple device emulation, leading to a more effective and comprehensive bug hunting methodology. The problem exists because the intricate dance between guest and host, mediated by complex emulated hardware, presents numerous opportunities for misconfigurations, logic flaws, and memory corruption vulnerabilities that can lead to guest-to-host escapes.

Key Findings

▶ Watch: Key tips for VMware hypervisor reverse engineering (2:50)

The speakers' extensive research into VMware's virtualization architecture yielded several key findings and methodological insights crucial for effective bug hunting. Central to their discoveries was the identification of the VMX binary as the single most important component in VMware Workstation and ESXi. This binary is responsible for initiating the virtual machine, and critically, it houses the vast majority of the virtual device emulation code. A significant finding was that the VMX binary is largely identical across both Workstation and ESXi, differing primarily in the system APIs it utilizes for host interaction. This implies that vulnerabilities discovered in one product often have direct applicability to the other, streamlining the research effort.

Their methodology emphasized starting the reverse engineering process from the hypervisor loop – the core mechanism that handles the frequent switching between the guest virtual machine and the host environment. By tracing execution from this fundamental loop, researchers can better understand the flow of control and data. They highlighted that since VMX operates as a user-mode process, it must interact with a kernel module for essential hypervisor operations. On Windows, this module is VMX86, while on ESXi, VMX communicates directly with the VMkernel. Tools like API Monitor were found to be invaluable for analyzing this communication, revealing the intricate dance between user-mode VMX and the host kernel.

A critical architectural mechanism identified was User RPC (Remote Procedure Call). This mechanism is VMware's custom design for the Virtual Machine Monitor (VMM) to interact with the VMX process on the host. It functions akin to a hypercall, but operates within user space, facilitating communication between the VMM and VMX. User RPC contains a substantial amount of code related to device emulation and is initiated with a single argument: a user RPC block pointer. This pointer directs to a shared area memory region, which represents another fundamental finding.

The Shared Area is a crucial mechanism for sharing data between the VMX and VMM. Its implementation is deeply intertwined with the initialization process of the VMM and its loading into memory. Understanding the creation, mapping, and usage of these shared memory regions is paramount, as they often serve as conduits for guest-controlled data to influence host-side processing. The speakers stressed the importance of studying the entire initialization process and paying close attention to every memory allocation and access pattern. Ultimately, their research reinforced the notion that while complex, virtual devices, such as those related to USB virtualization and SCSI virtualization, still represent the most "cost-effective" and fruitful targets for discovering vulnerabilities in virtualization environments.

Technical Deep Dive

▶ Watch: Locating the hypervisor loop and kernel module (4:05)

The technical foundation of VMware's virtualization architecture, as dissected by the speakers, revolves around several interconnected components, each presenting unique security research opportunities. The VMX binary stands as the central pillar of this architecture. As a user-mode process on the host, VMX is responsible for a multitude of critical functions, including the initial setup and launch of virtual machines, the execution of the hypervisor, and most importantly for security researchers, the emulation of virtual devices. The fact that VMX is largely identical between VMware Workstation and ESXi, differing only in its interaction with specific host system APIs, simplifies cross-platform vulnerability research. This implies that a bug found in Workstation's VMX often translates directly to ESXi's VMX, expanding the impact of a single discovery.

At the heart of the VMX's operation is the hypervisor loop, a continuous cycle that manages the transitions between the guest OS and the host. This loop is the entry point for many guest-initiated operations and a crucial starting point for reverse engineering. When a guest OS performs an action that requires hypervisor intervention—such as an I/O operation or a privilege change—the VMX process is invoked. Since VMX operates in user mode, it cannot directly access privileged hardware or kernel-level resources. This necessitates interaction with a kernel module. On Windows hosts, this interaction is facilitated by the VMX86 kernel driver. For ESXi, the VMX communicates directly with the VMkernel, which is the core operating system of the hypervisor itself. Analyzing this communication, often through tools like API Monitor, reveals the specific system calls and data exchanges that occur between VMX and the host's privileged components.

A critical inter-process communication mechanism is User RPC. This is a custom VMware design that allows the Virtual Machine Monitor (VMM) – the component that directly executes guest code – to communicate with the VMX process running on the host. Unlike traditional hypercalls that typically involve a direct transition from guest to hypervisor, User RPC operates at the user-space level, between the VMM and VMX. It is extensively used for initiating device emulation and other guest-requested operations. The invocation of a User RPC typically involves a single argument: a pointer to a user RPC block. This block resides within a specially designated shared area memory region.

The Shared Area mechanism is fundamental to the efficient operation of VMware virtualization. It represents a region of memory that is mutually accessible by both the VMM and the VMX process. This shared memory is crucial for transferring data, commands, and status information between the guest-facing VMM and the host-facing VMX. The implementation of this shared area is tightly coupled with the VMM's initialization and loading process. Understanding how these shared memory regions are allocated, mapped, and protected is paramount for security researchers. Flaws in managing these shared buffers, such as incorrect size checks, race conditions, or improper synchronization, can lead to critical vulnerabilities, including memory corruption or privilege escalation, as guest-controlled data flows into host-controlled parsing logic. The speakers emphasized the need to meticulously study the VMM's initialization routines, paying close attention to every memory allocation and access pattern, as these are often where the shared area is established and its security properties defined. By focusing on these architectural components and their interactions, the Tiangong team was able to identify vulnerabilities in complex virtual devices like USB and SCSI, demonstrating the efficacy of their holistic approach.

Demo / Proof of Concept

▶ Watch: Explanation of User RPC in VMware architecture (4:45)

The provided transcript mentions the speakers' intent to use "the bugs that we found in USB virtualization and SCSI virtualization to expand the possible source of 3D in virtual device." While the talk highlights the discovery of vulnerabilities in these specific virtual device categories and uses them as examples of fruitful bug hunting targets, the segment of the transcript provided does not detail a specific live demonstration or proof of concept for these findings. The focus in this portion of the talk is on the architectural understanding and methodology that led to such discoveries, rather than a step-by-step walkthrough of an exploit.

Defensive Implications

▶ Watch: Understanding the Shared Area mechanism for memory sharing (6:00)

The insights shared in "Bug Hunting In VMware Device Virtualization" offer crucial guidance for defenders seeking to harden virtualized environments. The primary defensive implication is the reinforced understanding that virtual devices remain a critical and high-value attack surface. Organizations running VMware Workstation or ESXi must prioritize the security of these emulated components, recognizing that flaws in their implementation can lead to guest-to-host escapes, compromising the entire host system and potentially other virtual machines.

Defenders should focus on several key areas:

1. Patch Management: The most fundamental defense is to ensure all VMware products, including Workstation and ESXi, are kept meticulously up-to-date with the latest security patches. Vulnerabilities in virtual devices, as well as the VMX and VMM components, are frequently discovered and addressed by vendors. Delayed patching leaves systems exposed to known exploits.
2. Architectural Understanding: Security teams should gain a deeper understanding of the VMware virtualization architecture, specifically the interaction between the VMX process, the VMM, and the host's kernel (VMX86 on Windows, VMkernel on ESXi). Understanding how User RPC and Shared Area mechanisms facilitate communication and data exchange is vital. This knowledge helps in identifying potential weak points, understanding the impact of reported vulnerabilities, and designing more robust monitoring solutions.
3. Input Validation and Fuzzing: The emulation of virtual devices involves parsing guest-controlled input. Defenders should assume that all input originating from a guest OS is potentially malicious. While this is primarily a vendor responsibility, understanding the types of vulnerabilities (e.g., memory corruption, integer overflows, logic bugs) that arise from insufficient input validation in device emulation can inform security audits and threat modeling. Organizations could consider internal fuzzing efforts on critical virtual device interfaces, where applicable and feasible, to proactively identify issues before adversaries do.
4. Least Privilege for VMX: Although VMX runs as a user-mode process, its interaction with kernel modules and its role in handling sensitive guest data means its privileges should be as restricted as possible. While this is largely controlled by VMware, understanding the security context of VMX can help in detecting anomalous behavior or potential compromise.
5. Monitoring and Detection: Implement robust monitoring solutions that can detect unusual activity related to VMX processes, kernel module interactions, or shared memory access patterns. Anomalies in CPU usage, memory consumption, or unexpected process behavior associated with VMX could indicate an attempted or successful exploit.
6. Isolation and Segmentation: Employ strong network and resource segmentation for virtual machines, especially those running untrusted workloads. While architectural bugs can bypass isolation, minimizing the blast radius of a successful escape remains a critical defense.

By proactively addressing these points, organizations can significantly enhance their defensive posture against the sophisticated threats targeting virtualized environments, moving beyond reactive patching to a more informed and resilient security strategy.

Key Takeaways

• Holistic Research Approach: Effective virtualization bug hunting requires understanding the entire architecture (VMX, VMM, kernel interactions, shared memory) rather than just individual virtual device code.
• VMX is Central: The VMX binary is the most critical component for VMware security research, being largely identical across Workstation and ESXi and containing most virtual device emulation logic.
• Virtual Devices Remain Key Attack Surface: Virtual devices, exemplified by USB and SCSI virtualization, continue to be the most "cost-effective" targets for discovering critical guest-to-host vulnerabilities.
• User RPC and Shared Area are Critical Interaction Points: VMware's User RPC mechanism and Shared Area memory regions are fundamental for VMM-VMX communication and data exchange, representing prime targets for security analysis and potential exploitation.
• Leverage Existing Resources: Successful reverse engineering and bug hunting benefit significantly from utilizing public resources such as CVEs, research papers, device documentation, and even QEMU code for comparative analysis.
• Prioritize Patching and Architectural Understanding: Defenders must prioritize timely patching of VMware products and cultivate a deep understanding of the underlying virtualization architecture to effectively mitigate risks and detect anomalous activity.

About the Speaker(s)

The talk was presented by JiaQing Huang and Hao Zheng, both security researchers with the Tiangong team at Qianxin Group. They are keenly interested in reverse engineering and virtualization security, having started their journey in this domain at the end of 2022. Despite their relatively recent entry into the field, their expertise is evident through their achievements, which include successfully escaping from Parallel Desktop at GeekCon 2023 and responsibly reporting numerous bugs in VMware Workstation and ESXi to the vendor. Their team leader, Yue Liu, also contributes to the Tiangong team, which consists of many individuals passionate about virus security. The team maintains a blog where they share their research findings, encouraging others to follow their work and engage with them for questions or discussions. Their work underscores a dedication to unraveling the complexities of virtualization to enhance its security.