SyzTrust: State-aware Fuzzing on Trusted OS Designed for IoT Devices

SyzTrust: State-aware Fuzzing on Trusted OS Designed for IoT Devices

Speakers: Qinying Wang; Boyu Chang; Shouling Ji; Yuan Tian; Xuhong Zhang; Binbin Zhao

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=TrmgaFyTA-0

Overview

The rapid proliferation of Internet of Things (IoT) devices has underscored the critical need for robust security mechanisms to protect sensitive data and operations. Trusted Execution Environments (TEEs) have emerged as a cornerstone of IoT security, providing an isolated and secure space for critical applications and data, shielded from potential attacks originating in the untrusted "normal world." However, the security of the Trusted Operating System (TOS), the foundational component within the TEE, is paramount; vulnerabilities in the TOS can compromise the entire secure environment, leading to system-wide crashes, data leakage, or even full system control by an attacker.

"SyzTrust: State-aware Fuzzing on Trusted OS Designed for IoT Devices" introduces a novel fuzzing framework specifically engineered to identify vulnerabilities in the resource-constrained and often closed-source TOSes prevalent in IoT devices. Presented by Qinying Wang from Zhejiang University and a team of researchers from Zhejiang University, EPFL, UCLA, and Georgia Tech, this work addresses significant challenges that have previously hampered effective security testing of IoT TEEs. By developing a hardware-in-the-loop design and pioneering state-aware fuzzing techniques, SyzTrust represents a significant leap forward in securing the foundational software of the interconnected world.

The importance of SyzTrust lies in its ability to uncover deep-seated vulnerabilities in critical security components that protect millions of smart devices, ranging from smart locks to FIDO security keys. Unlike previous fuzzing efforts that primarily targeted Android TEEs, SyzTrust tackles the unique complexities of IoT environments, which are characterized by severe resource limitations and hardware-dependent architectures. Its success in discovering 70 previously unknown vulnerabilities, leading to 19 high-severity CVEs in real-world IoT TOSes, underscores its immediate and profound impact on the security landscape of embedded systems.

Background

▶ Watch: Introduction to SyzTrust and IoT TE fuzzing challenges (0:00)

Trusted Execution Environments (TEEs) are an architectural feature designed to provide a secure, isolated environment for processing sensitive data and executing critical code. In the context of IoT devices, TEEs are essential for protecting assets like cryptographic keys, biometric data (e.g., fingerprints), and secure boot processes. A TEE introduces a conceptual division into a secure world and a normal world. Untrusted client applications and general-purpose operating systems (like Linux or RTOS) operate in the normal world, while trusted applications (TAs) and the Trusted Operating System (TOS) reside in the secure world. The TOS is the primary component within the secure world, offering cryptographic operations and other essential APIs for TAs to implement their functionalities.

Despite the critical role of the TOS in maintaining the integrity and confidentiality of sensitive operations, its security has historically been challenging to assess, especially in IoT contexts. Prior research has identified numerous vulnerabilities in TOSes, particularly those found in mobile phones, with one notable attack successfully extracting full disk encryption keys from Android's Key Master Services. However, these studies predominantly focused on Android TEEs, leaving a significant gap in the understanding and testing of security in IoT TEEs.

Fuzzing, an automated software testing technique that involves feeding invalid, unexpected, or random data to computer programs, has proven highly effective in discovering vulnerabilities. Yet, applying fuzzing to IoT TEEs presents unique and formidable challenges:

1. Hardware Dependency and Resource Limitations: IoT TOSes are far more hardware-dependent and resource-constrained than their mobile counterparts. This makes it difficult to instrument the code for coverage collection or to run even minimal fuzzing agents directly on the microcontroller. Many IoT TOSes are also closed-source and often encrypted, further hindering code instrumentation and execution monitoring within the secure world.
2. Statefulness of Operations: Trusted operating systems, especially for cryptographic or secure storage operations, must maintain complex internal states. For instance, an operation like TE_Allocate might initialize a cryptographic context, and only after this specific state is set can subsequent operations like T_SA_Update be triggered. Traditional coverage-based fuzzers, which primarily focus on code path coverage, often fail to explore these complex state transitions effectively. Without awareness of the internal state, such fuzzers struggle to reach deeper, state-dependent code paths, leading to missed vulnerabilities. The talk notes that some TOSes may need to maintain at least 20 states for sensitive operations.

These challenges highlight the necessity for a specialized fuzzing approach that can overcome the inherent limitations of IoT environments while effectively navigating the intricate state machine of trusted operating systems. SyzTrust was developed precisely to address these limitations, bridging the gap in automated security testing for this critical class of embedded software.

Key Findings

▶ Watch: Hardware-in-the-loop design for lightweight code coverage (4:00)

SyzTrust represents a significant advancement in the security testing of IoT Trusted Operating Systems, delivering several key findings and contributions:

1. First State-Aware Fuzzing Framework for IoT TEEs: SyzTrust is the pioneering framework designed specifically for fuzzing resource-constrained and hardware-dependent IoT Trusted OSes. It addresses the unique challenges of these environments, including closed-source codebases, limited resources, and complex state management.
2. Novel Hardware-in-the-Loop Design: The framework leverages ARM CoreSight Embedded Trace Macrocell (ETM) and a debug probe to stream real-time instruction traces from the secure world to an external computer. This hardware-in-the-loop approach offloads heavyweight tasks like coverage calculation and seed scheduling from the resource-limited IoT device, making fuzzing feasible.
3. Advanced Code and State Coverage Mechanisms:

• Lightweight Code Coverage: SyzTrust implements an event-based filter using data breakpoints to selectively collect ETM packets only during TOS execution of system calls, ignoring noisy traces from the normal world. It then calculates branch coverage directly from raw ETM packets, avoiding time-consuming instruction decoding.
• Sophisticated State Awareness: The framework infers critical state variables and handle structures by observing their values after specific API calls. It uses heuristics to identify non-random, operation-dependent state variables and monitors their values during fuzzing to calculate state coverage, guiding the fuzzer to explore deeper state transitions.

1. Superior Performance and Vulnerability Discovery: In evaluations against existing fuzzers like SyzCzar, SyzTrust significantly outperformed them in terms of both code coverage and state coverage. More importantly, it demonstrated its real-world impact by discovering 70 previously unknown vulnerabilities in multiple commercial IoT TOSes, including M-Tower from Samsung, Link-TEE from Alibaba Cloud, and T-TEE from S-Link Cloud.
2. High-Impact CVEs: Out of the discovered vulnerabilities, 19 have been assigned CVEs, all rated as high severity. These bugs encompass critical classes such as buffer overflows and null pointer dereferences, which can lead to severe consequences like system compromise, data leakage, or denial of service.
3. Open-Source Contribution: The SyzTrust framework and its code have been publicly released, enabling other researchers and developers to utilize and further enhance its capabilities, fostering a more secure IoT ecosystem.

These findings collectively demonstrate the effectiveness and necessity of SyzTrust's innovative approach in securing the often-overlooked and critically important Trusted Operating Systems in IoT devices.

Technical Deep Dive

▶ Watch: State-aware fuzzing: utilizing state variables for coverage (5:00)

SyzTrust's core innovation lies in its ability to overcome the two primary challenges of fuzzing IoT Trusted OSes: the difficulty of instrumentation and monitoring in resource-constrained, closed-source environments, and the need for state-aware exploration of complex API sequences. The framework is structured around a fuzzing engine and an execution engine, which communicate via a debug probe.

Addressing Instrumentation and Monitoring Challenges

The first major hurdle is obtaining meaningful code coverage from a secure world that is typically opaque and resource-limited. SyzTrust tackles this with a hardware-in-the-loop design leveraging specific hardware features:

1. ARM CoreSight Embedded Trace Macrocell (ETM): Modern ARM microcontrollers, common in IoT devices, often include an ETM. This component provides real-time instruction tracing, recording the execution path of the CPU. SyzTrust utilizes a debug probe (e.g., JTAG/SWD debugger) to stream these raw ETM packets from the target IoT device to an external computer. This offloads the heavy processing required for trace analysis from the device itself.
2. Event-Based Filtering: A significant challenge with ETM traces is their verbosity, as they record instructions from all execution contexts (client applications, normal world OS, trusted OS). To focus specifically on the TOS, SyzTrust implements an event-based filter. By setting two data breakpoints at the entry and exit points of TOS system calls, the hardware controller is configured to collect ETM packets only when the TOS is actively executing a system call. This dramatically reduces the "noisy" instruction traces from untrusted environments, ensuring that only relevant secure world execution data is captured.
3. Lightweight Branch Coverage Calculation: Rather than attempting to decode the complex ETM packets into instruction sequences and then aligning them with disassembled firmware (a time-consuming and error-prone process), SyzTrust proposes a more efficient method. It directly calculates branch coverage from the raw ETM packets. ETM packets inherently contain information about branch targets and taken/not-taken conditions. By processing these packets directly, SyzTrust can determine which branches have been taken, thereby inferring basic block coverage without the overhead of full instruction decoding and symbol resolution. This approach is crucial for performance and practicality in a hardware-in-the-loop setup.

This hardware-in-the-loop architecture allows the external computer (running the fuzzing engine) to handle all heavyweight tasks—coverage calculation, seed scheduling, and test case generation—while the IoT device is solely responsible for executing the test cases and streaming trace data.

Achieving State Awareness

The second critical challenge is the inherent statefulness of TOS operations. Traditional fuzzers often get stuck because they cannot transition between states required to trigger deeper code paths. SyzTrust introduces novel techniques for state variable inference and state coverage calculation:

1. Inferring State Variables and Handle Structures: The researchers observed that specific variables and handle structures within the TOS determine and store the operational state. For example, after a TE_Allocate operation (e.g., for a cryptographic context), an operation_state variable within a TE_Operation_Handler structure might change to indicate that a crypto operation is initialized. Only then can subsequent operations like T_SA_Update be triggered. To infer these critical state-defining variables, SyzTrust performs an initial profiling phase:

• It sends various TOS API sequences to the target and records the memory buffers corresponding to handle structures.
• It then applies heuristics to detect the adjusted ranges of state variables within these recorded buffers. The heuristics are based on two conditions: (1) the state variable is not randomly changeable, and (2) its value changes predictably according to different operation configurations (e.g., different API calls or parameters).

1. Monitoring and State Coverage: Once the address ranges of state variables are inferred, SyzTrust utilizes a monitor (likely integrated with the debug probe or via memory access features) to check the values of these state variables during the fuzzing process. State coverage is then calculated based on the unique combinations or transitions of these variable values. This allows the fuzzer to understand which internal states have been reached and which remain unexplored.

SyzTrust Fuzzing Architecture

The overall SyzTrust architecture integrates these components:

• Fuzzing Engine (Manager): Residing on the external computer, this component is responsible for:
• Test Case Generation: Generating sequences of TOS API calls and their parameters.
• Seed Preservation: Storing test cases that trigger new code paths or new states. It uses two maps:
• A hit map that stores the number of times each unique state has been hit.
• A seed map that stores "seed buckets" – collections of test cases – corresponding to each reachable state.
• Seed Selection: Strategically choosing the next test case to execute. It prioritizes states that are rarely hit (by consulting the hit map) to encourage exploration of less-traveled paths. Within the chosen state's seed bucket, it then selects a seed that achieves higher branch coverage, aiming for deeper code exploration within that state.
• Execution Engine: Residing on the IoT device, this component includes:
• Proxy Client Applications and Trusted Applications: These are implemented to execute the API sequences generated by the fuzzing engine. They act as the interface between the normal world and the secure world TOS.
• Hardware Controller: This component manages the debug probe and ETM, collecting the filtered code and state coverage information and streaming it back to the fuzzing engine.

The process is iterative: the manager generates a test case, sends it to the execution engine, which runs it on the device. The hardware controller collects coverage data (both code and state) and sends it back to the manager. The manager then uses this feedback to update its coverage maps, preserve valuable seeds, and intelligently select the next test case, continuously driving the fuzzer towards new and unexplored areas of the TOS.

The evaluation also considered the overhead of SyzTrust subprocesses. Results indicated that the test case execution itself consumed the majority of the time, while the orchestration and analysis components (coverage calculation, seed scheduling) incurred only about 1% of the overall time, demonstrating the efficiency of the hardware-in-the-loop and lightweight coverage approaches.

Demo / Proof of Concept

▶ Watch: Seed preservation and selection strategy (10:00)

While the talk did not feature a live, step-by-step demonstration of the SyzTrust framework in action, the researchers presented compelling evidence of its effectiveness through its application to real-world IoT Trusted Operating Systems. This served as the primary proof of concept, showcasing SyzTrust's practical utility and significant impact on device security.

The team applied SyzTrust to test three prominent commercial IoT TOSes:

1. M-Tower from Samsung: A widely deployed TEE solution in various Samsung devices.
2. Link-TEE from Alibaba Cloud: A TEE framework used in cloud-connected IoT devices.
3. T-TEE from S-Link Cloud: Another TEE solution likely targeting cloud-integrated embedded systems.

The results of these real-world applications were substantial:

• SyzTrust successfully discovered a total of 70 previously unknown vulnerabilities (zero-days) across these diverse TOS platforms.
• Out of these, 19 vulnerabilities were assigned CVEs (Common Vulnerabilities and Exposures), a testament to their severity and verifiable impact. All 19 CVEs were rated as "high variety" (high severity), indicating critical flaws that could lead to significant security breaches.
• The types of vulnerabilities identified included common but critical flaws such as buffer overflows and null pointer dereferences. Buffer overflows can lead to arbitrary code execution or denial of service, while null pointer dereferences often cause system crashes or information leaks, all of which are devastating in a secure environment.

Furthermore, the researchers compared SyzTrust's performance against SyzCzar, another fuzzer, specifically in terms of code coverage and state coverage. Their findings indicated that SyzTrust outperformed SyzCzar on both metrics, demonstrating the superior exploration capabilities of its state-aware and hardware-in-the-loop design. This comparative evaluation further validated SyzTrust's innovative approach as a more effective tool for discovering vulnerabilities in complex, stateful secure environments.

The successful discovery of numerous high-severity vulnerabilities in widely used commercial TOSes unequivocally serves as the proof of concept for SyzTrust's design and methodology, affirming its capability to enhance the security posture of the IoT ecosystem. The public release of the SyzTrust code further empowers the community to reproduce these findings and extend the framework's application.

Defensive Implications

▶ Watch: SyzTrust's performance and real-world vulnerabilities found (12:00)

SyzTrust's research highlights critical defensive implications for developers, vendors, and security practitioners involved with IoT devices and Trusted Execution Environments. The discovery of 70 zero-day vulnerabilities, including 19 high-severity CVEs, in commercial TOSes underscores that even foundational security components are not immune to flaws and require rigorous, specialized testing.

Here are key defensive implications:

1. Prioritize State-Aware Fuzzing for TOSes: Traditional fuzzing approaches, primarily focused on code coverage, are insufficient for stateful systems like Trusted Operating Systems. Developers and security teams must adopt or develop state-aware fuzzing methodologies to effectively explore the complex internal state machines of TOSes. SyzTrust provides a blueprint for how this can be achieved, by inferring state variables and guiding fuzzing based on state coverage.
2. Embrace Hardware-Assisted Security Testing: The challenges of instrumentation and monitoring in resource-constrained IoT TEEs necessitate innovative approaches. Leveraging hardware tracing capabilities like ARM CoreSight ETM and debug probes, as demonstrated by SyzTrust, is a viable and efficient strategy to gain visibility into secure world execution without impacting the target's performance or requiring source code access. Vendors should consider integrating such debug features into their development and testing pipelines, even if disabled in production.
3. Invest in Robust API Design and Input Validation: The prevalence of vulnerabilities like buffer overflows and null pointer dereferences indicates fundamental issues in how TOS APIs handle inputs and manage memory. Developers must meticulously design Trusted Application (TA) APIs with strict input validation, bounds checking, and secure memory allocation practices. Any data crossing the secure/normal world boundary must be treated with extreme suspicion and thoroughly sanitized.
4. Regular and Continuous Security Audits: Given the continuous evolution of attack techniques and the complexity of TOS codebases, one-time security audits are inadequate. IoT device manufacturers and TOS vendors should implement continuous security testing as part of their development lifecycle, integrating advanced fuzzing frameworks like SyzTrust (or similar internal tools) into their CI/CD pipelines to catch vulnerabilities early.
5. Contribute to and Utilize Open-Source Security Tools: The public release of SyzTrust's code is a significant contribution. Security researchers and developers should leverage such open-source tools to enhance their own testing capabilities. Contributing to these projects can foster a collaborative environment for improving IoT security globally.
6. Secure Supply Chain for TEE Components: As vulnerabilities in the TOS can compromise the entire device, it's crucial for IoT device manufacturers to demand rigorous security testing and certification from their TEE component suppliers. This includes requiring evidence of advanced fuzzing and vulnerability disclosure processes.
7. Patch Management and Update Mechanisms: Despite best efforts, vulnerabilities will still be discovered. IoT device manufacturers must ensure their devices have robust, secure, and user-friendly firmware update mechanisms that can deliver patches to the TOS effectively and reliably.

In essence, SyzTrust serves as a wake-up call, demonstrating that sophisticated, targeted fuzzing is not just possible but essential for securing the foundational software of the IoT. Defenders must integrate these advanced testing methodologies into their security strategies to build truly resilient IoT ecosystems.

Key Takeaways

• SyzTrust is the first state-aware fuzzing framework specifically designed for resource-constrained and hardware-dependent IoT Trusted Operating Systems (TOSes).
• It overcomes the challenges of closed-source environments and limited resources through a novel hardware-in-the-loop design utilizing ARM CoreSight ETM and debug probes for efficient trace collection.
• SyzTrust pioneers state-aware fuzzing by inferring critical state variables and calculating state coverage, enabling it to effectively explore complex, stateful API sequences in TOSes.
• The framework successfully discovered 70 previously unknown vulnerabilities (zero-days) in real-world commercial IoT TOSes, including Samsung's M-Tower and Alibaba Cloud's Link-TEE.
• These discoveries led to 19 high-severity CVEs, encompassing critical flaws like buffer overflows and null pointer dereferences, demonstrating SyzTrust's significant real-world impact on IoT security.
• SyzTrust's code has been open-sourced, providing a valuable tool for the security community to enhance testing and secure the IoT ecosystem.

About the Speaker(s)

The talk was presented by Qinying Wang from Zhejiang University. The project, SyzTrust, is a collaborative effort involving students and professors from several prestigious institutions: Zhejiang University, EPFL (École polytechnique fédérale de Lausanne), UCLA (University of California, Los Angeles), and Georgia Tech (Georgia Institute of Technology). The listed co-authors include Boyu Chang, Shouling Ji, Yuan Tian, Xuhong Zhang, and Binbin Zhao, indicating a multi-institutional research team with expertise in systems security, embedded systems, and fuzzing techniques. This collaborative background underscores the depth of research and diverse perspectives brought to the SyzTrust project.