eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System

eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System

Speakers: R. Sekar; Hanke Kimm; Rohit Aich

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=nxBhIxIRI9w

Overview

In the realm of cybersecurity, Advanced Persistent Threats (APTs) represent a formidable challenge. These sophisticated attack campaigns are characterized by their ability to bypass preventative security measures, establish long-term presence within enterprise systems, and remain undetected for extended periods, often weeks or months. Effectively combating APTs necessitates robust post-attack detection and forensic analysis capabilities. The talk "eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System," presented by Hanke Kimm, R. Sekar, and Rohit Aich from the Secure Systems Lab at Stony Brook University, addresses a critical gap in this defense strategy: the inadequacy of existing audit data collection systems.

The core problem lies in the inability of current logging solutions to reliably capture comprehensive system activity, particularly system call (syscall) logs, which are considered the "gold standard" for providing full provenance of an attack chain. Existing systems suffer from high overhead, significant data loss, and vulnerabilities to log tampering, rendering them ineffective for real-world APT detection and forensics. eAUDIT proposes a novel approach leveraging eBPF (extended Berkeley Packet Filter) to overcome these limitations, offering a system that is fast, scalable, and resilient against the very attacks it aims to detect.

This presentation details the development of eAUDIT, a system designed to eliminate data loss, drastically reduce overhead, minimize log tampering windows, and produce significantly smaller audit records. By combining innovative techniques such as compact encoding, a two-level buffering design, and an analytical performance model, eAUDIT sets a new benchmark for audit data collection. Its implications are profound, promising to empower defenders with the complete, untampered visibility required to detect and respond to even the most elusive APTs.

Background

▶ Watch: Introduction to eAUDIT and existing audit log problems (0:00)

The foundation of effective APT detection and forensic analysis rests squarely on the availability of high-fidelity audit data. Among various forms of audit logs, system call logs stand out as the most crucial, offering unparalleled provenance by connecting every piece of an attack campaign. These logs record every interaction between user-space applications and the operating system kernel, providing a granular view of system activity essential for understanding malicious behavior. However, the reliance on existing audit data collection systems has proven to be a significant vulnerability.

The speakers identify two primary drawbacks of current audit logging solutions:

1. High Overhead: Widely used systems, such as the Linux auditd logger system, impose substantial performance penalties. Experiments conducted by the eAUDIT team demonstrated that auditd can slow down workloads eightfold. This extreme overhead is not merely an inconvenience; it indicates that these logging systems are fundamentally unable to keep pace with the high rates of system calls generated by modern applications, leading directly to the second drawback.
2. Large Data Volume and Data Loss: The inability to keep up with system call rates forces existing loggers to drop records. The presentation highlighted alarming statistics: even at moderate loads, commercial and experimental audit logger systems drop a majority of events. Under more intense loads, data loss can exceed 90%, effectively rendering the collected data useless as critical attack events are simply not recorded. Furthermore, the sheer volume of data generated by traditional logging can overwhelm storage and processing capabilities.

These inherent weaknesses create powerful attack vectors for adversaries:

• Forcing Data Loss (Denial of Service): An intruder can deliberately generate intense amounts of benign system activity, effectively conducting a denial-of-service (DoS) attack against the logging system itself. By flooding the system with legitimate-looking events, the attacker can obscure their malicious actions, causing critical records to be dropped and leaving no trace of their activity.
• Log Tempering Vulnerability: This attack exploits the "window" of records that reside in the logger's memory before being transmitted to a secure, off-host server. If an attacker gains root access to the compromised system, they can exploit this window to erase audit records. The presentation revealed that existing systems can maintain log temper windows ranging from 50 to 500,000 audit records. Such a large buffer provides ample opportunity for an attacker to wipe out all indications of privilege escalation or other critical break-in activities before they are permanently stored.

To illustrate these points, the speakers presented empirical data. Using Postmark, a widely used file system benchmark, they showed that existing auditing systems indeed drop a majority of events, with some experiencing over 90% data loss at intense loads. The log temper windows were equally concerning, demonstrating that if a logger is killed, an attacker could easily remove evidence of compromise. This dire situation underscores the urgent need for a new audit collection system that is resilient to both data loss and log tampering attacks, a need that eAUDIT aims to fulfill.

Key Findings

▶ Watch: Introducing eBPF for building a new audit system (4:30)

The eAUDIT project successfully addresses the critical shortcomings of existing audit data collection systems, delivering a solution that is robust, efficient, and secure. The key findings and contributions of this work can be summarized as follows:

• Elimination of Data Loss: eAUDIT effectively solves the pervasive problem of data loss in audit logging. By implementing a combination of optimizations, the system ensures that virtually all system call events are captured, even under intense workloads where previous systems dropped over 90% of records.
• Substantially Reduced Data Volume: Through a highly efficient compact encoding scheme, eAUDIT produces event records that are significantly smaller than those generated by prior works. The average record size is reduced from over 170 bytes to approximately 17 bytes, leading to a 10-fold reduction in data volume. This not only eases storage burdens but also improves transmission efficiency.
• Dramatic Reduction in Overhead: The system achieves an exceptional reduction in performance overhead. Compared to an unoptimized eBPF-based design, the fully optimized eAUDIT design decreases overhead by a factor of 18 times. Across all benchmarks and loads, eAUDIT maintains an impressive 3% overhead, making it practical for deployment in high-performance enterprise environments without significantly impacting system operations.
• Minimized Log Tempering Window: eAUDIT drastically shrinks the window during which an attacker could potentially erase audit records. While systems like Sysdig produce log tampering windows ranging from 35,000 to 200,000 records, eAUDIT consistently maintains a window of only a couple hundred records on average. Even at maximum benchmark loads, the average window size is only about 1,000 records, making it hundreds of factors smaller than the next best performing system and significantly harder for an attacker to exploit.
• Novel Techniques for Overcoming Bottlenecks: The success of eAUDIT is attributed to several innovative techniques:
• A compact encoding scheme for syscall arguments.
• A two-level buffering design utilizing per-CPU caches to reduce ring buffer contention.
• An analytical performance model that optimally tunes latency and throughput, allowing for a balanced trade-off between system overhead and the log tempering window.

In essence, eAUDIT redefines the capabilities of audit data collection, providing a fast, scalable, and deployable system that is resilient to the data loss and log tampering attacks that plague conventional solutions.

Technical Deep Dive

▶ Watch: First optimization: compact encoding for reduced record size (6:10)

Building a resilient and high-performance audit data collection system presents significant technical challenges. The primary hurdles include the inherent difficulty of developing, debugging, and maintaining kernel code, and the necessity to handle extremely high rates of system calls, which can reach tens of millions of calls per second. eAUDIT addresses these challenges through a sophisticated architecture centered around eBPF and a series of critical optimizations.

Leveraging eBPF

The cornerstone of eAUDIT's design is eBPF (extended Berkeley Packet Filter). eBPF is a virtual machine implemented directly within the Linux kernel, allowing for the compilation and loading of safe kernel extensions without requiring modifications to the kernel source code. This eliminates the traditional complexities and risks associated with kernel development. For system call logging, eBPF is ideally suited because it can intercept every system call entry and exit. While eBPF programs typically have read-only access to kernel data, this is entirely sufficient for the purpose of collecting audit logs, as the goal is to observe and record, not modify, system behavior.

Naive eAUDIT Design and Initial Challenges

A straightforward eBPF-based architecture, which the speakers refer to as a "naive design," would involve an eBPF module capturing system call arguments from the Linux kernel and immediately sending these records to a user-level process, which then writes them to a file. For initial testing, this design intercepted approximately 80 Providence-related system calls, with record sizes comparable to previous systems (around 200 bytes).

While this naive eAUDIT design showed substantial improvement over traditional loggers, reducing data loss for the Postmark benchmark from over 60% to less than 20%, it was still insufficient. Intense loads could still push data loss to very high levels, demonstrating that eBPF alone, without further optimization, could not fully solve the data loss problem. The primary issue was identified not necessarily as the size of individual records, but the sheer rate of messages being enqueued onto the eBPF ring buffer, leading to contention.

Optimization 1: Compact Encoding Scheme

The first optimization introduced was a compact encoding scheme. Recognizing that system call arguments often contain redundant or verbose information, eAUDIT was designed to log only the essential system call identifier and its arguments in a highly compressed format. This strategy dramatically reduced the average record size from approximately 170+ bytes to about 17 bytes. This 10-fold reduction in record size directly translated to a decrease in data loss, improving performance by 10-20% across most benchmarks. While significant, the speakers noted that record size was not the primary bottleneck, but rather the frequency of interactions with the ring buffer.

Optimization 2: Per-CPU Cache (Two-Level Buffering Design)

The core issue identified was the contention on the eBPF ring buffer caused by needing to access it on every single system call. Despite the ring buffer's capacity for large data volumes, the high rate of individual messages created a bottleneck. The solution implemented was a per-CPU cache, forming a two-level buffering design.

Each CPU is equipped with its own dedicated cache. These caches hold messages up to a predefined size, P, which represents the number of system call events. Instead of sending each system call event individually to the ring buffer, events are first buffered in the per-CPU cache. Only when a cache fills up (i.e., P events have been collected) is its entire contents flushed and written as a single, larger message to the ring buffer. This strategy drastically reduces the effective system call rate to the ring buffer by a factor of P.

This optimization had a dramatic effect on overhead, reducing it from up to 100% down to single digits. For instance, setting P to 100 allowed eAUDIT to achieve overheads below 5% for all benchmarks. Crucially, by using a sufficiently high P value, this per-CPU caching mechanism effectively solved the data loss problem, ensuring that all system call events could be captured.

Optimization 3: Performance Model for Balancing Overhead and Latency

While larger P values increase throughput and reduce overhead, they also increase the number of records buffered in the per-CPU caches, thus increasing latency and, consequently, the log tampering window. To address this critical trade-off, eAUDIT incorporates a performance model designed to optimally balance overhead and latency.

The model considers two key parameters:

• P: The message cache size (number of system call events).
• W: The interval at which the user-level process is woken up to receive messages from the ring buffer.

The goal of the model is to find an optimal trade-off by minimizing the product of overhead (O) and latency (L). By differentiating O * L with respect to W and setting the result to zero, the team derived a formula for calculating the optimal value for W.

An experimental validation of this performance model demonstrated its accuracy, with measured overhead closely matching the model's predictions. The model indicated that an optimal W value typically falls between 5 and 8, representing the sweet spot where overhead is minimized without unduly increasing latency.

By combining all three optimizations—compact encoding, per-CPU caching, and the performance model—eAUDIT achieves a comprehensive solution. The final system demonstrates an 18-fold decrease in overhead compared to the unoptimized design, maintaining a consistent 3% overhead across all benchmarks and loads, while also keeping the log tampering window to a practical minimum of a few hundred records.

Demo / Proof of Concept

▶ Watch: Dramatic overhead reduction and solving data loss with caching (7:50)

While the presentation did not feature a live, interactive demo in the traditional sense, the speakers provided extensive experimental validation of eAUDIT's performance model and its overall efficacy. This validation served as the core proof of concept, demonstrating how the system overcomes the critical limitations of existing audit loggers.

The experimental validation focused on two main aspects:

1. Performance Model Accuracy: A key component of the proof of concept was illustrating how well their analytical performance model matched actual measurements on experimental platforms. Charts were presented showing measured overhead versus overhead predicted by the model. The data points clustered tightly around a diagonal line, indicating a near-perfect match between prediction and reality. This validated the theoretical underpinnings of their optimization strategy, particularly for finding the optimal user-level wakeup interval (W). The curves illustrating W's effect on overhead and latency further supported the theoretical finding that an optimal W is around 5 to 8, leading to minimal overhead.
2. Overall System Performance and Security Metrics: The most compelling part of the proof of concept was the presentation of eAUDIT's final performance and security metrics, achieved by combining all three optimizations:

• Data Loss Elimination: The system was shown to successfully eliminate data loss, capturing all events even under intense loads.
• Reduced Data Volume: The compact encoding scheme resulted in event records 10 times smaller than previous works.
• Minimal Overhead: The combined optimizations achieved a remarkable 3% overhead across all benchmarks and loads, a significant improvement (18 times lower) compared to the unoptimized eBPF design.
• Minimized Log Tempering Window: The most striking security improvement was in the log tampering window. eAUDIT consistently produced windows of only a couple hundred records on average. This was compared directly to the next best performing system, Sysdig, which exhibited windows ranging from 35,000 to 200,000 records. Even at maximum benchmark loads, eAUDIT's window only reached an average of about 1,000 records, making it hundreds of factors smaller and significantly more resilient to log tampering attacks.

These comprehensive experimental results and validations served as a robust proof of concept, demonstrating that eAUDIT is not just a theoretical improvement but a practical, high-performance, and secure audit data collection system ready for deployment.

Defensive Implications

▶ Watch: Third optimization: performance model for balancing overhead and latency (8:40)

The advent of eAUDIT offers profound defensive implications for organizations struggling with Advanced Persistent Threats and the limitations of conventional security monitoring. The ability to collect complete, untampered system call logs fundamentally changes the landscape of detection and forensics.

Here's what defenders should do with this information:

• Prioritize Comprehensive Syscall Logging: Defenders must recognize that partial or lossy audit logs are a critical blind spot. eAUDIT demonstrates that 100% system call capture without significant overhead is achievable. Organizations should evaluate their current logging solutions for data loss and consider adopting technologies like eAUDIT that guarantee full fidelity.
• Enhance APT Detection Capabilities: With full provenance provided by complete syscall logs, security teams can develop and deploy far more effective APT detection mechanisms. Machine learning models and behavioral analytics tools can now operate on a complete dataset, reducing false negatives and enabling the identification of subtle, multi-stage attack patterns that would otherwise be obscured by missing data.
• Strengthen Forensic Analysis: In the event of a breach, eAUDIT ensures that forensic investigators have access to every single system call executed during the compromise. This eliminates the guesswork and uncertainty caused by dropped records, allowing for more accurate root cause analysis, impact assessment, and the reconstruction of attacker timelines. This level of detail is invaluable for understanding how an attacker gained entry, what actions they performed, and what data they accessed or exfiltrated.
• Mitigate Log Tampering Risks: The drastically reduced log tampering window (down to hundreds of records compared to hundreds of thousands) is a game-changer. Defenders should incorporate this metric into their security assessments for logging solutions. Deploying systems like eAUDIT significantly raises the bar for attackers, making it exceedingly difficult to erase traces of their activity even after gaining root privileges. This enhances non-repudiation for audit trails.
• Evaluate Overhead and Scalability: The minimal 3% overhead and efficient data volume (10x smaller records) make eAUDIT suitable for deployment across a wide range of production systems, including those with high performance requirements. Defenders should no longer accept high overhead as an unavoidable cost of comprehensive logging. This enables broader deployment of robust logging across the enterprise.
• Explore eBPF for Security Observability: The success of eAUDIT highlights the power of eBPF as a secure and efficient mechanism for kernel-level observability. Security teams and architects should explore eBPF-based solutions for other security monitoring needs, recognizing its potential to provide deep insights without compromising system performance or stability.
• Consider Open Source and Community Contributions: The speakers encourage trying out their system. Defenders should explore the open-source availability (if applicable) of such solutions or advocate for their integration into commercial security products.

In summary, eAUDIT provides the critical visibility and resilience that has long been missing from audit data collection. By adopting these principles and technologies, defenders can significantly improve their ability to detect, analyze, and respond to the most sophisticated threats.

Key Takeaways

• Existing audit log collection systems suffer from critical flaws: high overhead (e.g., Linux auditd causes 8x slowdown), massive data loss (over 90% at intense loads), and large log tampering windows (50k-500k records).
• These flaws enable powerful attacks, including denial-of-service against loggers to obscure activity and log tampering by root attackers to erase evidence before logs are secured.
• eAUDIT leverages eBPF to safely and efficiently intercept system calls within the Linux kernel, overcoming the challenges of traditional kernel development.
• The system employs three key optimizations: a compact encoding scheme (reducing record size from 170+ to ~17 bytes), a per-CPU cache (two-level buffering) to minimize ring buffer contention, and an analytical performance model to optimally balance overhead and latency.
• eAUDIT achieves exceptional results: eliminating data loss, reducing data volume by 10 times, achieving a mere 3% overhead across all benchmarks (18 times better than unoptimized eBPF), and maintaining a log tampering window of only a couple hundred records (hundreds of times smaller than Sysdig).
• This breakthrough enables comprehensive, high-fidelity system call logging, providing defenders with the full provenance needed for robust APT detection, accurate forensic analysis, and significantly enhanced resilience against log tampering.

About the Speaker(s)

The presentation "eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System" was delivered by Hanke Kimm, R. Sekar, and Rohit Aich. They are researchers affiliated with the Secure Systems Lab at Stony Brook University. Their lab is dedicated to the investigation of Advanced Persistent Threats (APTs), focusing on understanding how these sophisticated attack campaigns manage to bypass preventative measures and remain undetected for extended periods. Their work directly contributes to developing better after-the-fact detection mechanisms and forensic analysis tools, with audit logs, particularly system call logs, being a central component of their research.