Not a True Copy: An In Depth Look at a Common Backup Format

Not a True Copy: An In Depth Look at a Common Backup Format

Speakers: Ron Brash

Conference: S4

YouTube: https://www.youtube.com/watch?v=RkkqAKAyrVU

Overview

In his S4 conference talk, "Not a True Copy: An In Depth Look at a Common Backup Format," Ron Brash unveiled a surprising and critical discovery regarding the integrity of widely used backup solutions. The presentation challenged the fundamental assumption that backups are exact, forensically sound copies of original data. Brash detailed a project initially aimed at helping a client, ABC Inc., address pragmatic security concerns with offline images, such as vulnerability scanning, malware detection, and creating Software Bill of Materials (SBOMs) for compliance. This seemingly straightforward task quickly escalated into an extensive reverse engineering effort that exposed significant discrepancies in how a prominent, unnamed backup vendor's solution handled data.

The core revelation of the talk was that these proprietary backups, despite being marketed as reliable recovery mechanisms, were "the same but different" from the original sources. This finding carries profound implications for incident response, legal culpability, and the overall trustworthiness of an organization's most critical data resilience strategy. Brash's work underscores a widespread oversight in the cybersecurity community: while frameworks and compliance mandates often stipulate the existence and testing of backups, there's a critical lack of scrutiny into the underlying formats and the integrity guarantees offered by the backup solution providers themselves. The talk served as a stark warning to defenders to move beyond superficial testing and deeply evaluate the forensic soundness of their chosen backup technologies.

Background

▶ Watch: Introduction: The hidden risks of proprietary backups (0:00)

The genesis of this eye-opening project stemmed from a customer, anonymized as ABC Inc., who approached Ron Brash's team with a set of seemingly reasonable and pragmatic objectives. ABC Inc. sought to enhance their security posture by analyzing offline system images for vulnerabilities, gaining intelligence on embedded components, detecting potential malware within their distribution processes, and ensuring attestation between their "SAT and FAT" (likely referring to System Acceptance Testing and Factory Acceptance Testing) and golden rolling images. A significant driver was the desire to avoid culpability in the event of a ransomware incident or other cyber events, particularly given their role as contractors deploying large machine cells. They also needed to generate SBOMs for compliance. Crucially, ABC Inc. was not permitted to install endpoint agents on these systems, necessitating an offline analysis approach.

However, as Brash humorously noted, projects rarely unfold as initially envisioned. This particular undertaking veered sharply when the team realized the fundamental question: "Does anyone actually evaluate the backup formats and the quality of the backup solution providers themselves?" The industry largely treats backups and recovery as "table stakes," with numerous solutions available and virtually every security framework, including ISA/IEC 62443, emphasizing their importance. Yet, evaluation often stops at cursory tests or compliance checks. Brash highlighted the lack of deeper scrutiny, such as tabletop exercises that might reveal critical gaps like missing license keys, or a thorough assessment of the solution's inherent quality and integrity.

The talk briefly touched upon the evolution of backup technologies, from the perceived "good times" of Windows XP and Windows NT backups to the changes introduced with Windows Vista. It also contrasted the proprietary solution under investigation with more transparent alternatives like Clonezilla (GPL, open source, supports many file systems) and historical tools like Norton Ghost. Microsoft's WIM (Windows Imaging Format) was also mentioned as an example of a format that has evolved to support Linux, highlighting the different mechanisms and compression techniques available. The proprietary vendor in question, however, offered flashy features, VM export functionality, centralized server models, cloud solutions, and standalone tools, but critically, their backups relied on "unsupported and unknown formats" – a black box that defied easy inspection and, as it turned out, true fidelity.

Key Findings

▶ Watch: Human condition of trust and backup format history (2:40)

The central and most impactful discovery of Ron Brash's project was that the proprietary backup format, used by several top vendors, produced images that were "the same but different" from the original data. This seemingly subtle distinction carries monumental implications. When the team attempted to validate the integrity and completeness of the backups against the original drives, they found inconsistencies at a fundamental level. The backups were not true, bit-for-bit copies, nor were they forensically sound representations of the original file systems.

This finding directly challenges the core assumption that organizations hold about their disaster recovery capabilities. If a backup is not an exact replica, its utility for several critical functions is severely compromised:

1. Legal and Forensic Admissibility: For investigations, legal proceedings, or compliance audits, the integrity and chain of custody of data are paramount. If a backup cannot be proven to be an exact, unaltered copy of the original, its forensic value is diminished, potentially rendering it inadmissible or unreliable as evidence. Brash explicitly stated that this particular vendor's backup format was not forensically sound.
2. Trust in Recovery: The ultimate purpose of a backup is reliable recovery. If the backup introduces subtle alterations or omissions, an organization cannot fully trust that a restored system or data set will function identically or possess the exact same characteristics as the original. This introduces an unacceptable level of risk, especially in critical infrastructure or industrial control systems (ICS) environments.
3. Vulnerability and Malware Scanning: The initial goal of ABC Inc. was to scan offline images for vulnerabilities and malware. If the backup process itself alters the image, it could either mask existing threats or introduce new inconsistencies, making accurate threat detection significantly more challenging, if not impossible.
4. SBOM Generation and Attestation: Generating accurate SBOMs requires precise knowledge of all components. If the backup process modifies files or structures, the resulting SBOM derived from the backup might not reflect the true state of the original system, undermining attestation efforts.

The "same but different" revelation forces a re-evaluation of how organizations perceive and validate their backup strategies, moving beyond mere functionality testing to a deeper interrogation of data integrity at the format level.

Technical Deep Dive

▶ Watch: Reverse engineering methodology for the proprietary format (4:00)

The journey to uncover the integrity issues in the proprietary backup format was a rigorous and extensive reverse engineering effort, described by Brash as an "ad hoc exploratory process." The team embarked on this endeavor after realizing that conventional methods, like using standard file clone tools, failed to yield the expected results – "Obi file Kenobi here, he didn't find the file clones... Just didn't happen that way."

The technical methodology involved several critical steps:

1. NTFS Volume Work from Scratch: A significant challenge was the lack of readily available tools or comprehensive documentation for deep analysis of NTFS volumes, particularly in the context of proprietary backup formats. Brash highlighted that "There's not that much code out there. There's not that much documentation that does it well." Consequently, the team had to develop their own capabilities for analyzing and understanding NTFS structures from the ground up, a substantial undertaking that formed the bedrock of their reverse engineering efforts.

1. Scalable Data Set Generation: To effectively identify discrepancies, the team created diverse data sets based on scalable sizes. The hypothesis was that differences might be subtle on small volumes but would become more pronounced and harder to manage as the volume size increased. This involved:

• Snapshots and Virtual Drives: Creating various snapshots and virtual drives of systems.
• Bit-Level Comparison: Performing meticulous bit-level comparisons between the original drives and their corresponding backup images.
• Test Files Exceeding Block Size: Utilizing carefully crafted test files designed to exceed the file system's block size. This was crucial for observing how the backup solution handled file fragmentation, movement, and sectioning, which could reveal underlying data manipulation or storage mechanisms.

1. Iterative Analysis and Validation: The process involved a systematic comparison workflow:

• Original Drive Analysis: Thoroughly walking through the original drives to understand their structure and content.
• Backup Image Inspection: Examining the backup images for completeness, looking for specific markers, and validating the presence and integrity of their test files.
• Assembly and Contrast: Assembling the reconstructed data from the backup and contrasting it against the original.
• Delta Analysis: Meticulously analyzing the deltas or differences between the original and the "recovered" data. This step was where the crucial "these were not the files we were looking for" realization occurred, indicating that the backups were not true copies.

1. Identifying Format Structures: While the specific proprietary format details were anonymized, Brash provided a glimpse into the types of structures they sought to reverse engineer:

• Magic Bytes: The team looked for basic structural elements like "magic" bytes, which are common identifiers at the beginning of file formats.
• Footers and Metadata: They discovered that critical metadata might be located in a "footer in reverse," requiring a rewind mechanism to access. This complexity added to the challenge of parsing the format.
• Split Archives: The format was identified as a "split archive," meaning large backups were segmented into multiple volumes. Brash showed an example of a "two of nine" marker for a specific volume. This splitting mechanism itself introduced potential points of failure or inconsistency, with issues tied to "thresholds" (auto or preferred splitting) and even "bugs" in the implementation.

The cumulative effect of these technical challenges and the observed discrepancies led to the definitive conclusion that the backup format was fundamentally altering the data, making it unsuitable for applications requiring forensic exactitude.

Demo / Proof of Concept

▶ Watch: Critical discovery: Backups are 'the same but different' (4:48)

While Ron Brash's talk did not feature a live, interactive demonstration of a tool or an exploit, the entire narrative served as a comprehensive proof of concept for their reverse engineering methodology. The detailed account of their "ad hoc exploratory process," including the development of custom NTFS analysis capabilities from scratch, the creation of scalable data sets, and the meticulous bit-level comparisons, effectively demonstrated the feasibility and necessity of such deep dives into proprietary backup formats.

The project itself, culminating in the discovery that a widely used backup solution produced "same but different" copies, stands as evidence that these fundamental integrity issues exist and can be uncovered through dedicated, expert analysis. The "Obi file Kenobi" analogy highlighted the initial failure of conventional approaches, underscoring that a deeper, more technical investigation was required to prove the lack of true fidelity in the backups. Thus, the talk's technical deep dive into how they uncovered these issues effectively functioned as the demonstration of their investigative capabilities and the veracity of their findings.

Defensive Implications

▶ Watch: Call to action: Rethink disaster recovery strategies (6:00)

The findings presented by Ron Brash carry profound implications for organizations across all sectors, particularly those operating critical infrastructure or handling sensitive data. Defenders must fundamentally shift their perspective on backup solutions from a mere compliance checkbox to a critical component of their forensic readiness and overall resilience strategy.

Here are key defensive implications:

1. Beyond Compliance Testing: Deep Integrity Validation: Organizations must move beyond superficial backup tests that merely confirm successful restoration. Instead, they need to implement rigorous, deep integrity validation processes. This involves:

• Bit-level Comparison: Where feasible, perform bit-level comparisons between original data and restored backups for critical systems and data sets.
• Checksum Verification: Actively verify checksums and cryptographic hashes across the entire data lifecycle, from creation to backup and restoration.
• Forensic Soundness Assessment: Explicitly assess whether backup solutions maintain forensic soundness. If a solution cannot guarantee an exact, untampered copy, its use for incident response, legal holds, or audits should be re-evaluated.

1. Scrutinize Backup Solution Providers: The talk highlights that "unsupported and unknown formats" are a significant risk. Defenders should:

• Demand Transparency: Insist on transparency from backup vendors regarding their underlying data formats, compression algorithms, and any data transformation processes.
• Ask Hard Questions: Inquire about how forensic integrity is maintained, whether backups are provably bit-for-bit copies, and what mechanisms are in place to detect alterations introduced by the backup process itself.
• Consider Open Formats: Evaluate backup solutions that utilize open, well-documented formats (like Microsoft WIM for Windows or open-source alternatives) where the integrity mechanisms are transparent and verifiable.

1. Enhanced Tabletop Exercises: Backup and recovery tabletop exercises need to be significantly more robust and realistic. Beyond just "can we restore?", exercises should include:

• Forensic Scenarios: Simulate scenarios where a restored backup needs to stand up to forensic scrutiny or legal challenges.
• Data Integrity Challenges: Introduce simulated data corruption or alteration to test the backup solution's ability to detect and recover unaltered data.
• Resource Availability: Test the availability of critical resources beyond just the backup data, such as license keys, specialized hardware, or expert personnel, as mentioned by Brash.

1. SBOMs and Attestation from Original Sources: If the integrity of backups cannot be guaranteed, organizations should prioritize generating SBOMs and performing attestation directly from the live systems or known-good golden images, rather than relying on potentially altered backup copies.

1. Understand the "Human Condition" of Trust: As Brash alluded to, there's a human tendency to trust that backups "are what we expect them to be." Defenders must challenge this complacency and foster a culture of skepticism and rigorous verification, acknowledging that even seemingly robust solutions can have hidden flaws.

By adopting these defensive postures, organizations can move towards a more resilient and forensically sound backup strategy, ensuring that their efforts to recover from cyber incidents are not undermined by the very solutions intended to protect them.

Key Takeaways

• Backups are Not Always True Copies: Organizations cannot assume that backup images are exact, bit-for-bit, or forensically sound copies of original data. Proprietary formats can introduce subtle but critical alterations.
• Deep Integrity Validation is Crucial: Beyond basic functionality testing, rigorous validation, including bit-level comparisons and checksum verification, is essential to confirm data integrity in backups.
• Forensic Soundness Matters: For incident response, legal, and compliance purposes, the forensic integrity of backups is paramount. If a backup is "the same but different," its value as evidence or for reliable recovery is compromised.
• Demand Transparency from Vendors: Organizations should scrutinize backup solution providers, demanding transparency about their underlying data formats, transformation processes, and explicit guarantees of data integrity.
• Rethink Tabletop Exercises: Backup recovery exercises need to evolve to include scenarios that test forensic soundness, data integrity challenges, and the availability of all necessary resources beyond just the data itself.
• Challenge Complacency: The "human condition" of trusting backups without deep evaluation is a significant risk. A culture of skepticism and thorough verification is necessary to ensure true resilience.

About the Speaker(s)

Ron Brash is a security researcher and expert who led the extensive reverse engineering project detailed in this talk. His work focused on uncovering vulnerabilities and integrity issues within proprietary backup formats used by major vendors. Through his team's efforts, he demonstrated the critical need for deeper scrutiny into backup solutions, challenging long-held assumptions about their reliability and forensic soundness. His presentation at S4 highlighted the practical challenges of analyzing opaque systems and the significant implications for industrial control systems (ICS) and critical infrastructure.