Detecting Ransomware Despite I/O Overhead: A Practical Multi-Staged Approach

Detecting Ransomware Despite I/O Overhead: A Practical Multi-Staged Approach

Speakers: Christian van Sloun, RWTH AR University

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=CMJoSHl5zdA

Overview

Ransomware continues to pose a significant threat to cybersecurity, consistently ranking among the top concerns for organizations worldwide. Attacks like the 2021 Colonial Pipeline incident underscore the profound real-world consequences, from widespread fuel shortages to significant economic disruption. This talk, presented by Christian van Sloun from RWTH AR University, delves into the critical challenge of detecting cryptographic ransomware—malware that encrypts user files and demands a ransom for their release—in real-time environments. While extensive research has focused on improving ransomware detection mechanisms, a crucial bottleneck has largely been overlooked: the substantial IO behavior monitoring overhead incurred by current detection systems, particularly on modern, high-performance storage devices.

The core problem addressed is that existing behavior-based ransomware detection, which relies on observing file access patterns, file types targeted, and other IO-related features, introduces significant performance penalties. On contemporary Solid State Drives (SSDs), this overhead can degrade performance by as much as 75%, effectively reducing an SSD's speed to that of an older hard disk drive (HDD). Christian van Sloun and his team propose an innovative multi-staged approach designed to dramatically reduce this monitoring overhead without sacrificing detection capabilities. By dynamically adjusting the level of scrutiny and feature monitoring based on a process's observed behavior, their system aims to make real-time ransomware detection practical and less intrusive on modern computing systems.

This research is particularly pertinent for security professionals, system administrators, and developers of endpoint detection and response (EDR) solutions. It provides a pathway to implement robust, behavior-based ransomware detection that can operate effectively in production environments without imposing unacceptable performance costs. The work highlights a critical engineering challenge in cybersecurity—balancing comprehensive security with system efficiency—and offers a well-researched, practical solution that moves the field closer to truly real-time, high-performance threat detection.

Background

▶ Watch: SSD performance degradation due to monitoring (2:15)

The concept of behavior-based ransomware detection is not new, with foundational research dating back to at least 2016. A seminal work in this area is the "Shield of S" paper by Continella et al., which introduced a self-healing file system. Their research revealed that simply creating a backup of a file upon write operations could introduce a performance overhead of 3.8 times. However, their real-world measurements suggested this overhead was not significantly noticeable to users. This apparent discrepancy, as van Sloun points out, was likely due to the prevalent use of hard disk drives (HDDs) at the time, which are inherently slow. The relatively minor IO overhead introduced by monitoring was masked by the HDD's baseline performance limitations.

The landscape has since dramatically shifted with the widespread adoption of Solid State Drives (SSDs). Modern SSDs boast significantly higher IOPS (Input/Output Operations Per Second) and throughput compared to HDDs. Consequently, the same IO behavior monitoring techniques that were inconspicuous on HDDs now cause substantial performance degradation on SSDs. Initial experiments conducted by van Sloun's team revealed that simply enabling the monitoring used in related work could degrade an SSD's sustained read and write performance by approximately 75%, making it perform like a traditional HDD. This stark reality led to the central question guiding their research: what causes this prohibitive overhead, how can it be reduced, and how does feature monitoring specifically impact it?

The underlying challenge lies within the operating system's IO stack. Applications do not directly access hardware; instead, they interact with files and devices through APIs provided by the operating system. This is where security monitoring typically takes place. On Linux, tools like eBPF (extended Berkeley Packet Filter) allow hooking into the virtual file system layer to observe and intercept IO operations. On Windows, mini-filters serve a similar purpose, enabling the insertion of custom code into the IO stack. Crucially, while security researchers are striving to monitor these IO operations for detection, OS developers are simultaneously working to optimize the IO stack for maximum efficiency to fully leverage modern high-speed SSDs. This creates a fundamental conflict of interest: security monitoring, by its nature, introduces additional processing steps into a system component that OS developers are meticulously refining for speed. The tension between comprehensive security visibility and peak system performance forms the bedrock of this research.

Key Findings

▶ Watch: I/O stack monitoring and efficiency conflict (3:10)

The research began by meticulously quantifying the sources of IO overhead. Initial measurements demonstrated that merely installing a driver that performs no monitoring (i.e., simply returns without executing additional code) introduces negligible overhead. The performance degradation only manifests once actual features begin to be recorded and parsed. For instance, simply extracting the process ID responsible for an IO operation already increases execution times for system calls. The overhead escalates significantly as more features are collected: resolving the full file name of an accessed file adds more processing, and calculating entropy within the kernel for read/write operations further exacerbates the slowdown. This established a clear correlation: the more features monitored, the greater the overhead and the slower the system's overall performance.

Recognizing that the majority of processes on a system are benign, the team proposed a multi-staged approach to intelligent monitoring. The core idea is to apply varying levels of scrutiny based on a process's current behavior. In a linear stage model, a process might start in "stage zero" with minimal feature monitoring, incurring very low overhead. As its behavior becomes more suspicious, it can be dynamically advanced to higher stages, where progressively more features are monitored, leading to increased scrutiny and, consequently, higher overhead. This dynamic adjustment allows the system to focus its resource-intensive monitoring efforts only on processes that exhibit potentially malicious characteristics.

The evaluation of this multi-staged approach yielded several critical findings regarding both detection performance and IO overhead. Using the Shield of S dataset—which contains real-world IO behavior—the researchers demonstrated that even with naive monitoring (where ransomware starts in the lowest scrutiny stage), the system successfully advances ransomware to a higher stage where it is ultimately detected. Furthermore, starting processes in an already high-scrutiny stage did not lead to an increase in false positives compared to traditional approaches, indicating the model's robustness.

A key achievement in terms of overhead reduction was observed for benign processes. Regardless of their initial stage, benign processes were quickly moved to stages with low monitoring overhead. Conversely, ransomware processes were rapidly advanced to stages with maximum scrutiny, enabling timely detection. This intelligent resource allocation led to a remarkable outcome: the average overhead on benign processes was reduced by one order of magnitude. This significant reduction holds true even when all processes are initially started in a stage with the maximum number of features. The researchers project that relaxing this initial assumption, allowing processes to start in a less scrutinized stage, would further decrease overhead. These findings validate the multi-staged approach as a viable strategy to achieve high detection performance while drastically mitigating the performance impact on modern systems.

Technical Deep Dive

▶ Watch: Feature monitoring impact on system overhead (4:30)

The proposed multi-staged approach is a sophisticated mechanism designed to balance rigorous ransomware detection with minimal system performance impact. At its heart is a linear stage model, where processes are dynamically assigned to different stages, each characterized by a distinct level of monitoring scrutiny and a corresponding set of features.

Stage Design Considerations:

The design of these stages and the transitions between them are governed by three critical properties:

1. Classification Performance: The primary goal is high recall across all stages, ensuring that ransomware is detected. However, a key insight of the multi-stage model is that precision can be relaxed in lower stages. A false positive in an early stage merely escalates a benign process to a higher scrutiny stage, resulting in temporary increased overhead, but not an immediate false alarm or system disruption. This flexibility allows for broader, less resource-intensive filtering in initial stages.
2. Classification Frequency: This refers to how often the system makes a decision about a process's behavior and potentially moves it between stages. If monitoring is disabled for certain calls in a lower stage, the system might miss crucial IO behavior that would trigger a re-evaluation. Therefore, intelligent triggers are needed to ensure timely re-classification.
3. Detection Latency: The number of stages directly impacts the time it takes to detect ransomware. More stages could mean a longer "critical path" from the start of ransomware execution to an alarm being raised, potentially allowing more damage to occur. The design must minimize this latency while still achieving overhead reduction.

The Linear Stage Model:

In the envisioned model, Stage 0 represents the lowest scrutiny, monitoring a minimal set of features (e.g., only process ID and basic call type) to maintain very low overhead. Subsequent stages (Stage 1, ..., Stage N) progressively increase the number of monitored features. Stage N, the highest scrutiny stage, monitors all available features, including computationally expensive ones like entropy calculation.

When an IO operation occurs, the system first determines the process's current stage. Based on this stage, a specific set of features is monitored and extracted from the IO call. This information is then pre-processed and fed into a classification model specific to that stage. The model makes a decision about the process's behavior (benign or suspicious). If the behavior warrants increased scrutiny, the process is moved to a higher stage; if it appears benign, it might remain in its current stage or even be demoted to a lower-scrutiny stage to further reduce overhead.

Evaluated Stage Designs:

The researchers explored four distinct stage designs:

1. Naive Monitoring: The baseline, where all features are monitored simultaneously regardless of process behavior. This represents the traditional approach with high overhead.
2. Simple Multi-Stage: Processes start in the lowest stage (minimal features) and are escalated based on suspicious activity.
3. Skipping Stages: A variant where processes can bypass intermediate stages if behavior is highly suspicious, aiming to reduce detection latency.
4. Higher Scrutiny Start: Processes begin in a higher-than-lowest stage, ensuring quicker detection but with potentially higher initial overhead for all processes.

Feature Monitoring and Overhead Quantification:

The team quantified the overhead caused by various features:

• Driver Installation Only: Negligible overhead.
• Recording Features (Process ID, Call Type): Measurable increase in execution times.
• Resolving File Names: Further increases overhead.
• Calculating Entropy (in-kernel): This feature, while crucial for detecting file encryption, is computationally intensive and contributes significantly to overhead.

The empirical measurements showed that monitoring all considered features simultaneously resulted in an average overhead of 180% on benign processes. Excluding entropy, which is a vital feature for ransomware detection, reduced this to 35%, still a substantial impact. This highlights the dilemma: essential detection features come with a high performance cost.

Detection Performance with Novel Ransomware:

To evaluate the efficacy of the multi-staged approach, the models trained on the 2016 Shield of S dataset were tested against novel ransomware samples from 2020-2023. The simple multi-stage model, without any modifications, was able to detect 12 out of 47 novel samples. This performance was on par with, and in some cases even outperformed, a single-stage approach using the same relatively old training data. Manually adjusting the hyperparameters slightly improved detection, leading the researchers to conclude that the multi-stage architecture itself does not inherently degrade detection performance; rather, the age of the training data (IO behavior patterns from 2016) was the primary limiting factor for detecting newer ransomware variants. This confirms the architectural soundness of the multi-staged approach from a detection perspective.

Overhead Reduction for Benign Processes:

The most compelling technical finding was the dramatic reduction in overhead for benign processes. Simulations demonstrated that benign processes, regardless of their starting stage, were rapidly moved to stages with minimal feature monitoring. Conversely, ransomware processes were quickly escalated to stages with maximum scrutiny. This intelligent resource allocation allowed the system to reduce the average overhead on benign processes by one order of magnitude. This was achieved even under the conservative assumption that all processes initially start in a stage with the maximum number of features. The implication is profound: real-time, behavior-based ransomware detection can be implemented with minimal impact on normal system operations, making it a truly practical solution for modern high-performance computing environments. The ability to dynamically adapt monitoring levels based on observed behavior is the cornerstone of this significant performance improvement.

Demo / Proof of Concept

▶ Watch: Proposed multi-staged ransomware detection approach (4:40)

While the talk does not describe a live, real-time demonstration of the system in action, the researchers emphasize that their methodology was rigorously evaluated. They refer to "artifacts which were evaluated," indicating a practical implementation and testing phase. The evaluation primarily relied on the Shield of S dataset, which contains real-world IO behavior, allowing for accurate measurements of both IO overhead and false positives under various monitoring scenarios.

The team performed detailed simulations to quantify the impact of their multi-staged approach. They measured the actual execution times of thousands of IO calls on a modern system, first establishing a baseline without any monitoring drivers, and then observing the increase in execution time under different feature combinations and multi-stage configurations. This systematic measurement approach allowed them to precisely determine the slowdown caused by their proposed system compared to an unmonitored baseline.

The speaker also touched upon deployment considerations, noting that while the evaluation was primarily on bare metal, they also used a virtual machine for artifact evaluation. They observed that while the increase in overhead was still present in a virtualized environment, the specific measurements and improvements were "widely different" compared to a bare metal approach due to the additional overhead introduced by virtualization itself. This suggests that while the principles hold, exact performance figures would need re-evaluation in specific virtualized deployments. Although a live PoC was not presented, the detailed evaluation framework and quantitative results serve as a strong proof of concept for the feasibility and benefits of the multi-staged approach.

Defensive Implications

▶ Watch: Stage design considerations: performance, frequency, latency (6:00)

The research presented by Christian van Sloun has profound implications for cybersecurity defenders, particularly those responsible for endpoint protection and incident response. The central takeaway is that effective, behavior-based ransomware detection no longer needs to come at the cost of crippling system performance. This opens the door for more widespread and aggressive deployment of real-time monitoring solutions.

Here are key defensive implications:

• Feasible Real-time Detection on Modern Systems: Defenders can now consider implementing or demanding endpoint protection solutions that leverage comprehensive IO behavior monitoring on SSDs without fear of unacceptable performance degradation. The multi-staged approach enables high-fidelity detection without turning high-performance workstations or servers into sluggish machines. This is crucial for environments where performance is paramount, such as development workstations, high-transaction servers, or critical infrastructure.
• Intelligent Resource Allocation: The principle of intelligently adjusting monitoring scrutiny based on process behavior is a paradigm shift. Instead of a blanket approach, defenders can advocate for systems that prioritize resources, focusing deep inspection on suspicious activities while allowing benign processes to run with minimal impact. This smart resource management not only improves performance but also reduces the attack surface by making it harder for ransomware to hide within the noise of normal system operations.
• Reduced Detection Latency and Damage: While the multi-staged approach introduces some detection latency (the time before ransomware is fully identified), the research indicates that this damage is manageable. The speaker noted that ransomware might encrypt approximately 3 megabytes of data before detection. This provides defenders with a critical window to initiate automated responses, such as process termination, network isolation, or restoration from shadow copies, thereby minimizing the overall impact of an attack. This "fail-fast" approach, even if some initial damage occurs, is vastly superior to not detecting the attack at all or detecting it too late.
• Enhanced EDR/EPP Capabilities: Vendors of Endpoint Detection and Response (EDR) and Endpoint Protection Platforms (EPP) can integrate this multi-staged monitoring logic into their agents. This would allow them to offer more robust, behavior-based ransomware modules that are less intrusive than previous iterations. Such enhanced capabilities would empower organizations to move beyond signature-based detection and heuristic analysis to a more dynamic and adaptive defense posture.
• Focus on Feature Selection: The research highlights the varying overhead caused by different features (e.g., process ID vs. entropy calculation). Defenders and security architects should understand this trade-off to make informed decisions about which features are essential for their specific threat models and performance requirements. Customization of monitoring profiles based on asset criticality and expected threat vectors could further optimize both security and performance.
• Artifacts for Replication and Research: The availability of artifacts evaluated in the research is a significant contribution. It allows other researchers and security teams to replicate the findings, experiment with the approach, and potentially integrate it into open-source security tools or custom defense mechanisms, accelerating the adoption of these advanced techniques.

In essence, this work provides a practical blueprint for building more resilient systems against ransomware. By addressing the fundamental performance bottleneck, it empowers defenders to deploy sophisticated behavioral analysis without compromising the very systems they are trying to protect.

Key Takeaways

• IO Overhead is a Major Bottleneck: Traditional behavior-based ransomware detection incurs significant performance overhead on modern SSDs, degrading performance by up to 75% and making real-time monitoring impractical.
• Multi-Staged Approach Reduces Overhead: A dynamic, multi-staged monitoring system, which adjusts the level of feature scrutiny based on process behavior, dramatically reduces overhead for benign processes by an order of magnitude.
• Detection Performance Maintained: The multi-staged model effectively detects ransomware, even novel variants, with performance comparable to single-stage models, demonstrating that reduced overhead does not compromise detection capabilities.
• Entropy is Crucial but Costly: Features like in-kernel entropy calculation are vital for detecting encryption but are also significant contributors to monitoring overhead, highlighting the need for intelligent, context-aware monitoring.
• Manageable Detection Latency: While the multi-staged approach introduces some detection latency, the damage caused before detection (e.g., ~3 megabytes encrypted) is low, enabling timely mitigation actions.
• Practical for Modern Systems: This research makes real-time, behavior-based ransomware detection a practical reality for modern, high-performance computing environments, enabling more robust endpoint protection without crippling system performance.

About the Speaker(s)

Christian van Sloun is a researcher from RWTH AR University in Germany. His work focuses on addressing critical challenges in cybersecurity, specifically in the realm of ransomware detection. In this talk, he presented research conducted with his colleagues, aiming to overcome the performance limitations of behavior-based monitoring on modern systems. His academic affiliation with RWTH AR University underscores a commitment to rigorous, scientific approaches to solving complex security problems.