The XZ Backdoor Story: The Undercover Op That Set the Internet on Fire

The XZ Backdoor Story: The Undercover Op That Set the Internet on Fire

Speakers: Thomas Roccia, Senior Threat Researcher, Microsoft

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=hwuIb-Vv2Ew

Overview

The XZ Backdoor story represents one of the most sophisticated and alarming supply chain attacks ever uncovered, a testament to the persistent and insidious nature of modern cyber threats. Presented by Thomas Roccia, a Senior Threat Researcher at Microsoft, this talk delves into the meticulous discovery of a backdoor hidden within the widely used XZ Utils data compression library. This operation, described as an "undercover op" spanning nearly three years, aimed to compromise a fundamental component of the open-source ecosystem, potentially granting unauthorized remote access to millions of Linux systems via SSH.

Roccia emphasizes that this incident surpasses even renowned attacks like SolarWinds, NotPetya, or the 3CX supply chain compromise in its technical sophistication and the extensive social engineering involved. The talk details the extraordinary series of coincidences and the sharp investigative skills of Andres Freund, Roccia's Microsoft colleague, that led to the backdoor's detection before it could be widely exploited. This presentation serves as a critical examination of the vulnerability, designated CVE-2024-3094, and its profound implications for open-source security and the broader cybersecurity landscape.

Background

▶ Watch: Introduction: The XZ Backdoor, a unique sophisticated attack (0:00)

The XZ Utils package is a cornerstone of the open-source world, providing free and open-source software for lossless data compression using the LZMA algorithm and the XZ format. Maintained by Lasse Collin for over a decade, it is a trusted and ubiquitous component across numerous Linux distributions, relied upon for efficient data handling and optimization. Its widespread adoption made it an attractive target for a sophisticated supply chain attack.

The critical link that made XZ Utils a viable vector for a backdoor targeting secure remote access lies in its integration with OpenSSH. OpenSSH, the de facto standard for secure remote login, utilizes XZ compression to reduce the size of transferred data. This dependency meant that compromising liblzma, a library within the XZ Utils package, could potentially allow an attacker to interfere with or even hijack SSH authentication processes. The "undercover operation" aspect refers to a long-term, multi-year effort by the threat actor, identified by the pseudonym Jia Tan, to gain trust within the XZ project, eventually becoming a co-maintainer and subtly introducing malicious code. This patient and calculated approach highlights the evolving threat landscape, where attackers are willing to invest significant time and resources to compromise foundational software components.

Key Findings

▶ Watch: Andres Freund's investigation triggered by SSH failures (2:20)

The discovery of the XZ backdoor on Friday, March 29, 2024, at 8:51 AM UTC, began with an email from Andres Freund to the OSS security mailing list, titled "backdoor in upstream XZ liblzma leading to SSH server compromise." Thomas Roccia clarifies the prevalent internet "legend" surrounding the discovery, which often credits a mere 500-millisecond delay. While the delay was a symptom, the true trigger was more complex and indicative of Freund's meticulous debugging process.

Freund initially observed SSH login failures during routine testing. This prompted him to investigate further, where he then identified a "substantial CPU usage" during SSH operations. It was this CPU spike that led him to pinpoint a 500-millisecond delay, which he later correlated with the new XZ package versions. He described the entire discovery as a "bunch of coincidence," underscoring the fortuitous nature of its detection.

The sequence of Freund's investigation was critical:

1. SSH login issues: The initial red flag.
2. CPU usage spike: Investigation revealed an unusual amount of CPU being consumed by the SSHD process.
3. Liblzma correlation: Further tracing identified liblzma as the source of the unexpected CPU activity.
4. Specific setup for discovery: Freund's environment played a crucial role. He was using a Debian unstable distribution, which, at the time, was one of the few Linux versions affected by the backdoor. Crucially, he was also using Valgrind with a specific flag: -Fno-omit-frame-pointer. Without this flag, Valgrind would not have "complained about the payload," providing the necessary diagnostic information. This enabled him to trace the high CPU usage within the SSHD processes to a location "below getCPUid," a highly unusual and suspicious finding.

The backdoor itself was found in liblzma versions 5.6.0 and 5.6.1 of the XZ Utils package. It was introduced by the mysterious user Jia Tan in February 2024. Fortunately, at the time of discovery, the affected versions were primarily deployed in development or unstable branches of Linux distributions, limiting widespread exploitation. Red Hat promptly assigned the vulnerability CVE-2024-3094.

Technical Deep Dive

▶ Watch: Specific technical setup crucial for backdoor discovery (4:00)

The XZ backdoor, designated CVE-2024-3094, represents a highly sophisticated method of compromising a critical open-source component. The malicious code was not directly inserted into the main source code but was instead hidden within obfuscated test files (bad-3-lzma-files) that were then integrated into the build process for the liblzma library. This technique ensured that the malicious payload was only compiled under specific conditions, making it harder to detect through casual code review.

The core of the backdoor's functionality revolved around injecting malicious code into the liblzma library, which is dynamically linked by various applications, including OpenSSH. When liblzma versions 5.6.0 or 5.6.1 were compiled on specific Linux distributions (like Debian unstable or Fedora Rawhide) and subsequently loaded by an SSH daemon (sshd), the backdoor would activate. The injected code was designed to intercept and modify the authentication process within SSH.

Andres Freund's critical observation of high CPU usage "below getCPUid" within the sshd process provided a key insight into the backdoor's operation. This indicates that the malicious code was likely hooking into low-level system functions or library calls, potentially manipulating the control flow or data processing related to authentication. The use of Valgrind with the -Fno-omit-frame-pointer flag was instrumental. This flag provides more detailed stack traces by preserving frame pointers, allowing Valgrind to detect unusual behavior or errors within the obfuscated payload that would otherwise be hidden. Valgrind's complaints about the payload under these specific debugging conditions were the direct indicators of the malicious code's presence and its attempt to execute stealthily.

The social engineering aspect of the attack, though not detailed in its technical execution within this talk, is a crucial part of the deep dive into its sophistication. The actor "Jia Tan" spent nearly three years contributing to the XZ project, building trust, and eventually gaining maintainer status. This allowed them to introduce the malicious changes without immediate suspicion, demonstrating an unparalleled level of patience and strategic planning from the attackers. The backdoor itself was designed to be highly conditional, activating only on specific system configurations (e.g., specific architectures, compiler flags, and library versions), further increasing its stealth and making it difficult to analyze without the precise setup Freund employed. This multi-layered approach—combining social engineering, obfuscated code injection, and conditional activation—underscores the advanced nature of this threat actor.

Demo / Proof of Concept

▶ Watch: Explaining XZ Utils package and mysterious Jia Tan's role (5:45)

During this presentation at DEF CON 32, Thomas Roccia focused on the detailed narrative of the XZ backdoor's discovery and its broader implications. No live demonstration or technical proof-of-concept of the backdoor's functionality or exploitation was performed or described in the transcript. The talk primarily served to educate the audience on the intricacies of how such a sophisticated supply chain attack was uncovered.

Defensive Implications

▶ Watch: Why XZ Utils is critical for many Linux distributions (6:30)

The XZ backdoor incident serves as a stark reminder of the critical vulnerabilities inherent in the open-source supply chain and necessitates a re-evaluation of defensive strategies. Defenders must recognize that even fundamental, widely trusted components can be targeted through sophisticated, long-term operations.

Firstly, immediate patching and downgrading are paramount. Any systems running XZ Utils versions 5.6.0 or 5.6.1 must be immediately downgraded to a known safe version, such as 5.4.x or earlier, or updated to patched versions released by distribution maintainers. System administrators should verify their XZ Utils package versions across all Linux environments, especially those exposed to the internet via SSH.

Secondly, the incident underscores the importance of proactive monitoring for anomalous system behavior. Andres Freund's discovery stemmed from observing SSH login failures and subsequent "substantial CPU usage" by the sshd process. Organizations should implement robust monitoring solutions that can detect unusual resource consumption, unexpected delays in critical services like SSH, or deviations from baseline behavior. Anomalies, no matter how subtle, should trigger thorough investigation.

Thirdly, the role of developer diligence and advanced debugging tools cannot be overstated. Freund's use of Valgrind with the -Fno-omit-frame-pointer flag was crucial. Defenders and developers should be encouraged to utilize similar deep inspection tools during testing and development, paying attention to any warnings or unusual outputs, even if seemingly minor. This level of scrutiny can help uncover obfuscated or maliciously crafted code that might evade standard static analysis.

Finally, this attack highlights the need for enhanced scrutiny of open-source project contributions and maintainer changes. The multi-year social engineering campaign to introduce "Jia Tan" into the XZ project reveals a vulnerability in how trust is established and maintained within open-source communities. While difficult to implement without stifling collaboration, discussions around more rigorous vetting processes for new maintainers or significant code changes in critical projects are essential. Organizations should also consider implementing software supply chain security practices, such as Software Bill of Materials (SBOMs), to track dependencies and their provenance, aiding in rapid identification of affected components during future incidents.

Key Takeaways

• Unprecedented Sophistication: The XZ backdoor represents one of the most sophisticated supply chain attacks, involving a multi-year social engineering campaign and highly obfuscated malicious code.
• Fortuitous Discovery: The backdoor was found due to a "bunch of coincidence" involving SSH login failures, unexpected CPU usage, and the use of specific debugging tools like Valgrind with the -Fno-omit-frame-pointer flag on a Debian unstable system.
• Critical Open-Source Vulnerability: The attack targeted XZ Utils and its liblzma component, a fundamental part of the Linux ecosystem, demonstrating the severe risk of compromising widely used open-source software.
• SSH Compromise Potential: The backdoor aimed to inject malicious code into the sshd process, potentially allowing unauthorized remote access and control over affected systems.
• Immediate Defensive Actions: Systems running XZ Utils versions 5.6.0 or 5.6.1 must be immediately downgraded or patched to mitigate the CVE-2024-3094 vulnerability.
• Enhanced Monitoring and Scrutiny: The incident underscores the critical need for vigilant system monitoring for anomalous behavior and increased scrutiny of open-source contributions and dependencies.

About the Speaker(s)

Thomas Roccia is a Senior Threat Researcher at Microsoft, where he played a key role in investigating the XZ backdoor campaign upon its discovery by his colleague, Andres Freund. Originally from France, he currently resides in Australia. Roccia is known for his work in threat research and shares his insights and findings on his website, www.securitybreak.io, and on X (formerly Twitter) under the handle @fr0gger_. His expertise lies in uncovering and analyzing sophisticated cyber threats, making him a valuable voice in the cybersecurity community.