Non-intrusive and Unconstrained Keystroke Inference in VR Platforms via Infrared Side Channel

Non-intrusive and Unconstrained Keystroke Inference in VR Platforms via Infrared Side Channel

Speakers: Tao Ni, City University of Hong Kong

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=wURSz8j6P4E

Overview

As virtual reality (VR) technologies become increasingly integrated into daily life, spanning immersive gaming, healthcare, online meetings, and industrial design, the security implications of these platforms are growing in significance. This talk, presented by Tao Ni from City University of Hong Kong, unveils a novel and critical privacy vulnerability in widely adopted VR devices. The research identifies an infrared (IR) side channel stemming from the core tracking mechanisms of VR controllers, which can be exploited to infer virtual keystrokes typed by users.

The presentation introduces VR-Key, an innovative, non-intrusive, and model-free framework designed to reconstruct unconstrained virtual keystrokes from the leaked IR signals. Unlike prior work that often relies on machine learning models or visual observations, VR-Key offers a robust, end-to-end solution that operates effectively even in challenging, low-visibility, or concealed attack scenarios. This research underscores the need for greater attention to hardware-level side channels in emerging technologies, highlighting how seemingly innocuous design choices for functionality can inadvertently expose sensitive user data.

The findings are particularly relevant given the widespread deployment of VR devices that utilize constellation tracking systems, such as the Meta Quest 2 and Pico 4. By demonstrating high success rates in character and word-level keystroke inference across various conditions, the talk presents a compelling case for re-evaluating the security posture of current VR platforms and developing proactive defensive measures.

Background

▶ Watch: Introduction: VR security and keystroke inference (0:15)

The rapid expansion of VR applications across diverse sectors, from entertainment to professional use, has brought with it an escalating need to address inherent security and privacy risks. Over the past five years, numerous potential side-channel attacks on VR devices have been documented, capable of causing severe privacy leakage. As society continues its digitalization, VR is poised to become an even more integral part of our lives, making the investigation of these vulnerabilities paramount.

A key design feature found in most commercial off-the-shelf VR devices, particularly those relying on handheld controllers, is a distinctive ring arc. While often perceived as an aesthetic element, this ring serves a crucial functional purpose: it houses multiple infrared (IR) LEDs. These IR LEDs are fundamental to the constellation tracking system, a mechanism that facilitates interaction between the VR headset and its controllers. The VR headset, equipped with an array of cameras, continually scans the IR signals emitted by these LEDs. This continuous communication enables the system to precisely track the user's body movements and hand gestures, which is essential for virtual interactions such as navigating menus or typing on a virtual keyboard.

However, this very mechanism, designed to enhance user experience and interaction fidelity, inadvertently creates a significant privacy loophole. The IR signals, while intended for internal system communication, are not confined within the device's operational boundaries; they are emitted into the ambient environment. This leakage of IR signals forms the basis of the side-channel vulnerability. Attackers, equipped with appropriate IR sensors, can passively capture these leaked signals without direct interaction with or modification of the target VR system. The principle is straightforward: any interaction that causes a change in the controller's position or state, such as pressing a virtual key, will alter the pattern or intensity of the emitted IR signals, creating a detectable fluctuation that can be intercepted and analyzed. This phenomenon exemplifies how new functionalities and hardware integrations, while advancing technology, can simultaneously introduce unforeseen security challenges.

Key Findings

▶ Watch: Introducing "VR Key": a model-free attack (2:55)

The research presented in this talk uncovers a critical and previously under-explored vulnerability in mainstream VR platforms, leading to several key findings:

• Discovery of a Novel IR Side Channel: The core finding is the identification of an infrared (IR) side channel in commercial VR controllers that utilize constellation tracking systems. This side channel arises from the leakage of IR signals emitted by the controllers' embedded IR LEDs, which are designed for positional tracking but inadvertently broadcast information to the surrounding environment.
• Development of VR-Key, a Model-Free Attack Framework: The talk introduces VR-Key, an innovative end-to-end framework for non-intrusive and unconstrained keystroke inference. A distinguishing feature of VR-Key is its model-free approach, meaning it does not rely on extensive training data or machine learning models, allowing it to reconstruct open-world keystrokes (i.e., any sequence of characters) without prior knowledge of the user's typing patterns or the virtual keyboard layout.
• High Effectiveness in Keystroke Inference: VR-Key demonstrated high attack success rates, achieving over 80% accuracy in inferring both character-level and word-level virtual keystrokes. This was validated on popular commercial VR devices, specifically the Meta Quest 2 and Pico 4, indicating a widespread vulnerability across devices employing similar tracking technologies.
• Robustness Across Diverse Conditions: The attack proved remarkably robust against various external factors and environmental conditions. This includes different user orientation angles relative to the sensor array, varying distances between the victim and the attacker's sensors (effective at 2 to 4 meters), different VR devices, and even variations in the design of the IR sensor array. Furthermore, it accounted for complexities like typing with single or multiple controllers and even user movements during typing, maintaining its effectiveness in most scenarios.
• Feasibility in Real-World Attack Scenarios: The research detailed three practical attack scenarios, showcasing the real-world applicability of VR-Key:

1. Concealed Attack: The IR sensor array could be hidden behind a glass door covered with a one-way film, making it undetectable by the victim.
2. Reflected IR Signals: The attack successfully captured and utilized IR signals reflected off common surfaces like glasses, demonstrating its ability to operate indirectly.
3. Low Visibility Attack: VR-Key proved effective in low-light conditions or at night, where traditional camera-based visual attacks would fail, highlighting its unique advantage.

• Proposed Countermeasures and Limitations: The talk also outlined potential defensive strategies, including IR encryption and shuffling virtual keyboards, while acknowledging the trade-offs (e.g., usability reduction). Importantly, the research identified that newer VR headsets like the Apple Vision Pro, which adopt 100% hand tracking and eye tracking modes without IR LEDs in controllers, would not be susceptible to this specific side channel, though most commercial VR devices still rely on the vulnerable controller-based IR tracking.

Technical Deep Dive

▶ Watch: Three real-world attack scenarios demonstrated (3:30)

The VR-Key attack framework is an ingenious four-stage process designed to reconstruct virtual keystrokes with precision and without reliance on machine learning models, a significant departure from many prior side-channel attacks. The core of its operation hinges on the precise capture and analysis of infrared (IR) signal fluctuations emitted by VR controller LEDs.

The attacker's setup involves a custom-built IR sensor array strategically placed within the vicinity of the VR user, typically within a range of 2 to 4 meters. This array is designed to detect the subtle changes in IR signal patterns that occur when a user interacts with a virtual keyboard. Each keystroke, or even the movement of a controller to hover over a key, causes a unique fluctuation in the IR signals as the controller's orientation and distance to the headset (and thus the attacker's sensors) change.

Stage 1: Coordinate Calibration

The first challenge in inferring keystrokes on a virtual keyboard is its dynamic nature. Unlike a physical keyboard, a virtual keyboard can float in 3D space and possess varying orientations. To overcome this, VR-Key employs a sophisticated coordinate calibration technique. The system leverages the variations in response time of IR signals captured by multiple sensors within the array during a single timeframe. By analyzing these temporal differences, the framework can accurately ascertain the orientation angles of the virtual keyboard in real-time. Once the orientation is known, the system calculates the precise coordinates and projects the potential typed keystrokes from the 2D plane of the IR sensor array onto the 3D virtual keyboard plane. This crucial step ensures that the detected IR events are correctly mapped to their corresponding keys on the virtual interface.

Stage 2: IR Feature Extraction

This stage is central to VR-Key's model-free methodology. Instead of training complex machine learning models, the system directly extracts meaningful features from the raw IR signal data. These features are analyzed in both the time domain and the frequency domain to characterize typing events. Time domain analysis focuses on the duration and intensity profiles of IR fluctuations, capturing the "press" and "release" events of a virtual key. Frequency domain analysis, on the other hand, helps identify unique patterns or signatures associated with different controller movements or key presses.

The extracted features are then used to generate weight-based confusion matrices. These matrices represent the likelihood of a specific IR signal pattern corresponding to a particular virtual key. By overlaying these matrices as heatmaps, the system visually and algorithmically determines the most probable typed virtual keystrokes. The talk mentioned the use of algorithms from OpenCV for data visualization and processing, indicating a robust computational approach.

A critical aspect addressed in this stage is handling situations where users might type using both VR controllers simultaneously. This can introduce "IR retentions" or ambiguous signals. VR-Key incorporates mechanisms to differentiate between signals originating from the left and right controllers, effectively removing these ambiguities to accurately determine which controller is responsible for a given typing event.

Stage 3: Keystroke Recovery

With the typing points and their probabilities established, the next stage focuses on reconstructing the actual keystroke sequence. This is achieved by logically connecting the identified "boxes" or regions of high probability on the virtual keyboard plane. The framework also analyzes the typing speed of different users. The research found that the average typing speed for most users was approximately 0.25 seconds per keystroke, a valuable metric for validating the reconstructed sequence and distinguishing intentional inputs from random noise. The sequential analysis of these connected points, combined with timing information, allows for the accurate inference of the typed characters.

Stage 4: Semantic and Grammar Inspection (Conditional LLM Usage)

To enhance the plausibility and accuracy of the reconstructed keystrokes, particularly for longer phrases or sentences, VR-Key incorporates a conditional post-processing step involving large language models (LLMs). If the reconstructed input is determined not to be a randomized password (e.g., a known pattern, a short, common phrase), the framework leverages a zero-shot prompt in large language models to inspect the semantic and grammatical correctness of the output. This step helps to refine the inferred keystrokes, correcting minor errors and ensuring the output is contextually and syntactically reasonable. However, for genuinely randomized passwords, the LLM is not used, preserving the integrity of potentially sensitive, non-linguistic inputs. This intelligent conditional application of LLMs demonstrates a nuanced understanding of attack scenarios and data types.

The comprehensive four-stage framework, from precise calibration to intelligent post-processing, allows VR-Key to achieve high accuracy in reconstructing virtual keystrokes without the traditional overhead of machine learning training, making it a highly effective and adaptable side-channel attack.

Demo / Proof of Concept

▶ Watch: Evaluation: high keystroke inference success rates (6:40)

The practical applicability and robustness of the VR-Key attack were rigorously demonstrated through a series of real-world evaluations and proof-of-concept scenarios. The researchers utilized two prevalent commercial off-the-shelf VR devices for their experiments: the Meta Quest 2 and the Pico 4. This choice of devices, both widely adopted in the consumer market, underscores the broad impact of the discovered vulnerability.

The core demonstration involved setting up a custom-built IR sensor array in a room where a user was interacting with a virtual keyboard within the VR environment. The array was designed to capture the subtle fluctuations in IR signals emitted by the VR controllers' LEDs as the user typed. The visual representation of these captured signals, showing clear fluctuations corresponding to typing events, served as an initial prototype demonstration of the IR leakage.

To highlight the practical threat, the attack was validated across three distinct real-world scenarios:

1. Concealed Attack: In this scenario, the IR sensor array was hidden behind a glass door, which was covered with a one-way film. This setup demonstrated that an attacker could covertly monitor a VR user without being detected, as the sensor array was visually obscured but still fully capable of capturing the leaked IR signals.
2. Reflected IR Signals Attack: This experiment showcased the ability of VR-Key to infer keystrokes even when direct line-of-sight to the controllers was obstructed. The IR sensor array successfully captured signals that had reflected off various surfaces in the room, such as windows or other reflective objects. This proves that the attack is not limited to direct observation but can exploit ambient reflections, making it harder to mitigate.
3. Low Visibility Attack: Addressing a limitation of many visual side-channel attacks, VR-Key was tested in conditions of low ambient light or complete darkness (e.g., at midnight). Since the attack relies on infrared, which is invisible to the human eye and unaffected by visible light conditions, it maintained high effectiveness. This scenario emphasized the unique advantage of the IR side channel where traditional camera-based surveillance would fail.

Beyond these specific scenarios, the researchers conducted extensive robustness testing to demonstrate VR-Key's resilience to various environmental and operational factors. The attack consistently achieved over 80% attack success rates in keystroke inference, even under conditions such as:

• Different orientation angles: The user and virtual keyboard could be oriented differently relative to the sensor array.
• Varying distances: The attacker's array was effective at distances ranging from 2 to 4 meters.
• Different VR devices: Success was consistent across both Meta Quest 2 and Pico 4.
• Different IR sensor array designs: Variations in the array's configuration did not significantly degrade performance.
• Single and multiple-source keystroke events: The system effectively handled inputs from one or both controllers.
• User movements: Even when users made slight movements while typing, the attack remained robust.

These comprehensive demonstrations and evaluations firmly establish VR-Key as a potent and practical threat, capable of extracting sensitive keystroke information from widely used VR platforms under a variety of challenging conditions.

Defensive Implications

▶ Watch: Proposed countermeasures: IR encryption and shuffling (7:50)

The discovery and demonstration of the VR-Key attack necessitate immediate attention from VR platform developers, hardware manufacturers, and users regarding the security of virtual environments. The implications are profound, as sensitive data like passwords, financial information, or private communications typed within VR could be compromised. The research proposes several countermeasures, though each comes with its own set of challenges and trade-offs.

1. IR Encryption and Protocol Redesign: The most direct and robust defense lies in modifying the fundamental communication protocol of the constellation tracking system. The proposed solution involves incorporating encryption schemes directly into the IR communication. By encrypting or dynamically modifying the IR patterns used for tracking, attackers would be prevented from deriving meaningful information from the leaked signals. This would require a redesign of the IR communication stack, ensuring that the emitted IR patterns are either randomized, obfuscated, or contain cryptographic primitives that make them useless to an eavesdropper without the corresponding decryption key. The talk indicated that early investigations showed this method to be effective, suggesting it's a viable long-term solution, albeit one requiring hardware and software updates from manufacturers.

1. Shuffling Keyboards / Signal Masking: A more immediate, albeit less comprehensive, defense involves user interface (UI) level mitigations. This includes implementing shuffling keyboards, where the layout of the virtual keyboard changes dynamically for each input session or even for each key press. This makes it significantly harder for an attacker to correlate detected IR fluctuations with fixed key positions. Similar to shuffling PIN pads on ATMs, this technique introduces randomness at the input layer. This method is particularly recommended for sensitive input fields, such as when entering bank account details or passwords.

Another related approach is signal masking, where the VR system actively introduces noise or irrelevant IR signals during sensitive inputs to obscure the actual typing events. However, a significant drawback of these UI-level mitigations, particularly shuffling keyboards, is a potential reduction in user usability. Constantly changing layouts can be disorienting and slow down typing, impacting the user experience. Therefore, a balance must be struck between security and usability.

1. Hardware-Level Attenuators or Filters: While not explicitly detailed in the talk, another potential hardware defense could involve physical modifications to the IR LEDs or controllers themselves. This might include using specialized coatings or filters that restrict the emission angle of the IR signals, ensuring they are primarily directed towards the headset's cameras and minimizing ambient leakage. However, such modifications would need to be carefully engineered to avoid impacting the legitimate tracking functionality.

1. User Awareness and Best Practices: For end-users, awareness of this side channel is crucial. While direct mitigation might be difficult, understanding that typing in VR can be monitored by an attacker with an IR sensor array in the same room might encourage more cautious behavior, such as avoiding the input of highly sensitive information in shared or public VR spaces, or using alternative input methods where available.

It's important to acknowledge the limitations highlighted by the researchers: devices like the Apple Vision Pro, which rely on 100% hand and eye tracking without IR LEDs in their controllers, are inherently immune to this specific IR side channel. However, the vast majority of current commercial VR devices still utilize controller-based IR tracking systems due to their perceived accuracy in interaction, meaning the vulnerability remains widespread. VR platform developers and hardware manufacturers must prioritize addressing this fundamental design flaw to safeguard user privacy in the evolving landscape of virtual reality.

Key Takeaways

• VR Controllers' IR LEDs Create a Side Channel: The core design of most commercial VR controllers, using infrared (IR) LEDs for constellation tracking systems, inadvertently creates a privacy-threatening side channel by leaking IR signals into the ambient environment.
• VR-Key Enables Model-Free Keystroke Inference: The VR-Key attack is a novel, non-intrusive, unconstrained, and model-free framework that can accurately infer virtual keystrokes from these leaked IR signals without requiring prior machine learning training.
• High Effectiveness on Popular VR Devices: The attack demonstrates high success rates, achieving over 80% accuracy in inferring both character and word-level keystrokes on widely used devices like the Meta Quest 2 and Pico 4.
• Robustness Across Challenging Conditions: VR-Key is highly robust, proving effective in challenging scenarios such as concealed attacks, utilizing reflected IR signals, low-visibility environments, varying distances (2 to 4 meters), and even with user movements or dual-controller typing.
• Defensive Measures Are Crucial: Mitigations include redesigning IR communication protocols to incorporate IR encryption and implementing UI-level defenses like shuffling virtual keyboards, though the latter may impact usability.
• Future-Proofing is Needed, Current Devices Vulnerable: While newer devices like the Apple Vision Pro are immune due to different tracking technologies, the majority of existing VR platforms still rely on the vulnerable controller-based IR tracking, necessitating urgent security enhancements.

About the Speaker(s)

The talk "Non-intrusive and Unconstrained Keystroke Inference in VR Platforms via Infrared Side Channel" was presented by Tao Ni from the City University of Hong Kong. His research focuses on identifying and analyzing security vulnerabilities in emerging technologies, particularly within the domain of virtual reality, to enhance user privacy and platform robustness. His work, as demonstrated in this presentation, highlights a deep technical understanding of hardware-software interactions and their security implications.