SmartCookie: Blocking Large-Scale SYN Floods with a Split-Proxy Defense on Programmable Data Planes

SmartCookie: Blocking Large-Scale SYN Floods with a Split-Proxy Defense on Programmable Data Planes

Speakers: Sophia Yoo

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=GbVLZhpY8o

Overview

In an era where cyberattacks are increasingly sophisticated and volumetric, the persistent threat of SYN flooding remains a significant challenge for network providers. Despite being a known attack vector since the mid-1990s, SYN floods are still ranked as the second most common form of Denial-of-Service (DoS) attack. In her talk at USENIX Security '24, Sophia Yoo presented "SmartCookie," a novel split-proxy defense designed to mitigate large-scale SYN flooding attacks by strategically leveraging programmable data planes.

SmartCookie addresses the limitations of traditional SYN flood defenses, which often fall short in terms of security, scalability, or performance. The core innovation lies in its collaborative, multi-layered architecture, which intelligently partitions defense functionalities across a high-speed P4 programmable switch in the network data plane and a lightweight eBPF component in the Linux kernel data plane. This design ensures that each component handles tasks for which it is uniquely suited, allowing SmartCookie to block massive volumes of attack traffic while maintaining low latency for legitimate client connections, even under extreme duress.

The significance of SmartCookie extends beyond merely blocking SYN floods; it proposes a blueprint for building robust, adaptive defenses against modern network threats. By demonstrating how programmable hardware and software can collaborate to overcome inherent limitations of each, SmartCookie offers a path forward for securing critical network infrastructure against evolving adversarial tactics. It represents a crucial step in the ongoing battle against DoS, providing a high-performance, scalable, and secure solution where previous approaches have struggled.

Background

▶ Watch: Smart Cookie: A split-proxy defense against SYN floods (0:00)

To appreciate the innovation behind SmartCookie, it's essential to understand the mechanics of a SYN flooding attack and the evolution of its canonical defense, the SYN cookie.

A SYN flooding attack exploits the fundamental three-way handshake process of TCP connection establishment. When a client initiates a connection, it sends a SYN packet to a server. The server responds with a SYN-ACK packet and, critically, allocates memory to store state information for the pending connection. The client is then expected to complete the handshake by sending a final ACK packet. In a SYN flood, an adversary sends a deluge of SYN packets with spoofed source IP addresses. The server, unaware of the malicious intent, dutifully allocates memory for each incoming SYN and sends SYN-ACKs. However, because the source IPs are spoofed or the adversary simply never sends the final ACK, these memory blocks remain allocated indefinitely, or until a timeout occurs. The server's memory resources are quickly exhausted, preventing it from accepting new, legitimate connections, thereby causing a denial of service. This is an asymmetric attack because the server expends more resources (memory allocation, SYN-ACK retransmissions) than the attacker (sending a single SYN packet).

The traditional defense against SYN floods is the SYN cookie mechanism, proposed in the mid-1990s. The core idea of SYN cookies is to trade memory resources for compute resources. Instead of allocating memory for a pending connection, the server cryptographically encodes all necessary connection state (such as the initial sequence number, maximum segment size, and a timestamp) into a "cookie" value. This cookie is then sent back to the client as the initial sequence number in the SYN-ACK packet. The server then forgets about the interaction, freeing up any potential memory allocation. If a legitimate client responds with an ACK packet containing the correct cookie value (derived from the SYN-ACK's sequence number), the server can reconstruct the connection state from the cookie, verify its authenticity, and proceed with establishing the connection. If the ACK is invalid or never arrives, no server memory was wasted.

While ingenious, the canonical SYN cookie defense, particularly when implemented purely in software on the server, faces significant challenges in modern network environments:

1. Computational Strain on Servers: Cryptographically generating and verifying cookies, along with packet processing, for every potential connection is computationally intensive. As attack volumes escalate, this computation becomes a new attack vector, overwhelming the server's CPU capacity. Servers, typically designed to handle megabit per second to low gigabit per second rates, can easily experience CPU exhaustion, leading to a denial of service for legitimate clients.

1. Limitations of Hardware-Only Proxies: To offload this computational burden, researchers explored moving the SYN cookie defense to dedicated hardware proxies, such as high-speed programmable switches, which can process traffic at terabit per second speeds. However, hardware-only solutions introduce their own set of problems:

• State Management Bottleneck: When a hardware proxy intercepts and handles SYN cookies, it effectively creates two connections: one between the client and the proxy, and another between the proxy and the server on behalf of the client. This necessitates header translation and requires the switch proxy to maintain per-flow state for all verified and ongoing benign connections. The limited memory of high-speed switch hardware can quickly become a bottleneck, particularly when scaling to hundreds of thousands of concurrent benign connections. While solutions like Jocken attempted to circumvent this memory usage by avoiding per-flow state, they did so at a significant performance penalty on all benign flows.
• Insecure Hashing: For SYN cookies to be secure, they must be generated using a strong cryptographic hash function. This hash typically takes as input the connection's 4-tuple (source IP, destination IP, source port, destination port) along with a secret value. A robust cryptographic hash prevents adaptive adversaries from manipulating hash collisions to bypass the defense. Unfortunately, many hardware-only solutions today rely on simple CRC checksums for hashing. CRC checksums are designed for error detection, not cryptographic security, rendering the cookie defense vulnerable to manipulation and making the entire system insecure.

These challenges highlight the critical need for a defense that is not only scalable and performant but also cryptographically secure, without overwhelming either server CPUs or switch memory.

Key Findings

▶ Watch: Challenges: Costly software cookie computation, CPU exhaustion (3:30)

SmartCookie's central contribution is its profound insight that modern SYN flooding defenses necessitate a layered, collaborative split-layer approach across both hardware and software targets. This architecture is "smart" because of its intelligent division of labor, meticulously designed from first principles to leverage the unique strengths and mitigate the inherent weaknesses of each processing environment.

The design process began by identifying the three key functional elements of any SYN cookie proxy defense:

1. Cookie Checks: This encompasses both the initial cryptographic generation of the cookie and its subsequent verification when a client returns an ACK.
2. Header Translations: Mapping connection details between the client-proxy and proxy-server connections.
3. State Keeping: Maintaining records for verified, ongoing connections.

With these elements identified, SmartCookie then addressed how best to partition these functionalities across different network components:

• Switches (Hardware): High-speed programmable switches are inherently excellent at processing massive volumes of packets at terabit-per-second rates. They excel at rapid, stateless packet forwarding and simple rule application. However, their memory resources are severely limited, making them unsuitable for maintaining large amounts of per-flow state for hundreds of thousands of concurrent connections. This critical insight positions switches as an ideal first line of defense for blocking the bulk of large-scale attacks, but not for detailed, long-term state management.

• Servers (Software): Servers, in contrast, are provisioned with ample memory to handle their benign flows and application-level state. However, their CPUs can be easily exhausted by high packet processing rates and complex cryptographic computations, especially under attack. This makes servers ideal for precisely tracking benign flows and handling the nuances of connection state, but not for bearing the brunt of high-volume attack traffic.

Armed with these insights, SmartCookie architects a defense that offloads the most resource-intensive and high-volume tasks to the hardware while delegating precise state management and false-positive handling to the software layer. This strategic division of labor ensures high performance, robust security, and unparalleled scalability, overcoming the limitations that plagued previous single-layer or poorly integrated hardware/software solutions. The result is a system that can effectively absorb and neutralize terabit-scale SYN floods without impacting the quality of service for legitimate users.

Technical Deep Dive

▶ Watch: Challenges: Hardware proxy state, memory limits, insecure hashes (4:30)

SmartCookie implements its split-proxy defense architecture through two primary agents: the SmartCookie Switch Agent and the SmartCookie Server Agent, complemented by a custom collaborative protocol. The system is designed to operate transparently, requiring no modifications to either the client's or the server's network stack.

1. SmartCookie Switch Agent (Hardware Layer):

• This agent runs on a high-speed P4 programmable switch, positioned as the first line of defense within the network. Its primary role is to shoulder the computational load of generating and verifying SYN cookies under immense traffic volumes.
• Secure Cookie Generation: Crucially, the switch agent generates cookies using a cryptographic hash function, taking the connection's 4-tuple (source IP, destination IP, source port, destination port) and a secret value as input. This directly addresses the security vulnerability of prior hardware solutions that relied on insecure CRC checksums, making SmartCookie robust against sophisticated adaptive adversaries attempting hash collision attacks.
• Approximate State Tracking: Recognizing the memory limitations of switches, the SmartCookie Switch Agent does not store exact per-flow state for verified connections. Instead, it tracks verified connections in an approximate manner using a variant of a Bloom filter. A Bloom filter is a probabilistic data structure that can test whether an element is a member of a set. While it can produce false positives (indicating an element is present when it's not), it never produces false negatives. By compressing multiple benign connections into this compact data structure, the switch can efficiently identify legitimate returning ACKs without exhausting its limited memory. When a client's ACK packet arrives with a valid cookie, and the Bloom filter indicates a high probability of a legitimate connection, the switch forwards the packet to the server.

1. SmartCookie Server Agent (Software Layer):

• This agent runs as a lightweight eBPF (extended Berkeley Packet Filter) component within the Linux kernel on the protected server. eBPF allows for safe, sandboxed execution of programs within the kernel, providing high performance and fine-grained network control without modifying the kernel source code.
• Exact State Records and False Positive Handling: The Server Agent is responsible for maintaining exact records of verified connections. While the switch uses an approximate Bloom filter, the server's ample memory is leveraged to store precise connection state. This is critical for supporting header translations between the proxy-server connection and the actual server application, which requires exact state information.
• The Server Agent also plays a crucial role in handling any false positives that might arise from the approximate tracking done by the Bloom filter in the switch. If the switch incorrectly identifies an attack packet as benign due to a Bloom filter collision and forwards it, the eBPF agent can detect this discrepancy using its exact state records and drop the malicious packet before it consumes server resources. This two-tiered verification ensures that only truly legitimate connections are established.

1. Custom Collaborative Protocol:

• A custom protocol orchestrates the interaction between the SmartCookie Switch Agent and the SmartCookie Server Agent. This protocol defines how connection state is implicitly or explicitly communicated and synchronized (or not synchronized, in the case of the approximate switch state) between the hardware and software layers. The talk refers to the paper for more details on this protocol, as well as the specifics of the robust hashing algorithms and the design of the approximate data structures.

This design ensures that the high-volume, initial filtering and secure cookie generation occur at line rate on the P4 switch, preventing attack traffic from ever reaching the server's CPU. Simultaneously, the server's eBPF agent handles the precise state management and final verification, ensuring accuracy and full functionality for legitimate connections, even when the switch operates with approximate state. This synergistic approach allows SmartCookie to achieve high security, scalability, and performance simultaneously.

Demo / Proof of Concept

▶ Watch: Smart Cookie's Key Insight: Layered hardware and software defense (6:18)

While the talk did not feature a live demonstration of SmartCookie in action, Sophia Yoo presented compelling performance evaluation results that serve as a robust proof of concept for its effectiveness, scalability, and efficiency. The evaluation focused on demonstrating SmartCookie's ability to maintain low latency for benign connections under various attack rates, highlighting its superior performance compared to existing solutions.

The key performance metric presented was the average end-to-end latency for a benign connection setup combined with a full HTTP request and response, measured in milliseconds. This was tested on machines within the same network, excluding external internet delays to isolate the system's performance.

The results were visualized with:

• X-axis: Attack rate, ranging up to millions of packets per second (mpps).
• Y-axis: Average end-to-end latency in milliseconds.

The evaluation showcased the following critical findings:

• Baseline Performance: A baseline measurement, representing the latency without any SYN cookie defense, was established at 1.08 milliseconds.
• SmartCookie's Near-Baseline Latency: SmartCookie, depicted in red, consistently "hugs" this baseline, demonstrating an impressive average latency of 1.71 milliseconds. This low overhead is maintained even as attack rates escalate significantly, indicating its minimal impact on legitimate traffic.
• Superiority over Jocken: In stark contrast, Jocken, a prior hardware-only solution designed to avoid memory bottlenecks, started with a significantly higher end-to-end latency of 11.12 milliseconds. This substantial penalty highlights the trade-offs in Jocken's design choices and underscores SmartCookie's ability to achieve memory efficiency without sacrificing performance.
• Exceptional Scalability: The evaluation also vividly demonstrated SmartCookie's scalability under extreme attack conditions:
• Both generic kernel-level SYN cookies and the Jocken solution exhibited severe performance degradation. Their latencies "shoot up" and they begin to drop benign connections at relatively low attack rates—around 3 mpps for kernel solutions and 36 mpps for Jocken. This indicates their capacity limits are quickly reached.
• SmartCookie, however, maintained zero packet loss until 136 mpps. Even beyond this point, it continued to support "reasonable client latency" at even higher attack rates, showcasing its remarkable resilience and ability to handle volumetric attacks far beyond the capabilities of its predecessors.

The talk emphasized that while the presentation focused on performance, the accompanying paper provides additional details and results concerning SmartCookie's security and scalability, confirming that it indeed delivers on all three fronts. The results clearly validate the effectiveness of SmartCookie's split-proxy, layered design in providing a robust, high-performance defense against large-scale SYN floods.

Defensive Implications

▶ Watch: Smart Cookie's intelligent division of labor across programmable targets (6:48)

SmartCookie presents a compelling case for a paradigm shift in how organizations approach Denial-of-Service (DoS) mitigation, particularly against SYN floods. The insights and architecture offered by SmartCookie have several critical defensive implications:

1. Embrace Programmable Data Planes: The most immediate implication is the demonstrated value of programmable data plane targets, specifically P4-enabled switches. Organizations should consider integrating these high-speed, flexible hardware platforms into their network edge or DDoS scrubbing centers. This allows for early-stage, line-rate filtering and attack mitigation, preventing malicious traffic from consuming upstream network or server resources.

1. Adopt Split-Layer Architectures: SmartCookie's success underscores the power of intelligently distributing defense mechanisms across different layers of the network stack. Instead of monolithic solutions, security architects should design defenses that leverage the strengths of each component: high-speed hardware for volumetric processing and approximate filtering, and flexible software (like eBPF) for precise state management and handling edge cases. This division of labor maximizes efficiency and resilience.

1. Prioritize Cryptographic Strength: The emphasis on using cryptographic hash functions for cookie generation, rather than simple checksums, is a vital lesson. Defenders must ensure that any SYN cookie implementation relies on strong cryptographic primitives to prevent adaptive adversaries from bypassing the defense through hash manipulation. Security should not be sacrificed for performance.

1. Leverage Kernel-Level Programmability (eBPF): The SmartCookie Server Agent highlights the significant potential of eBPF for enhancing host-based defenses. eBPF allows network engineers and security professionals to implement high-performance, custom packet processing logic directly within the Linux kernel without requiring kernel recompilations or sacrificing stability. This enables fine-grained control, efficient state management, and robust false-positive handling at the server level, acting as a crucial second line of defense.

1. Consider Approximate vs. Exact State: SmartCookie demonstrates the effectiveness of combining approximate state tracking (e.g., Bloom filters on switches) for high-volume, early-stage filtering with exact state tracking (on servers) for final verification and critical operations like header translation. This hybrid approach optimizes resource usage, allowing hardware to operate at line rate while ensuring correctness and security at the host.

1. Future-Proofing DDoS Defenses: The vision for SmartCookie's split-layer design to be applied to other protocols, attacks, and IDS/IPS systems suggests a broader strategy for network security. This architectural model provides a flexible framework that can be adapted to evolving threats, offering a more agile and robust defense posture against a wider range of attacks beyond just SYN floods. Organizations should consider how this layered approach can enhance their overall security infrastructure.

In summary, SmartCookie provides a practical and high-performing blueprint for modern DDoS defense. It encourages network operators and security teams to invest in programmable infrastructure and adopt sophisticated, multi-layered strategies that are both scalable and cryptographically sound.

Key Takeaways

• SYN flooding remains a prevalent and challenging DoS attack, necessitating modern, high-performance defenses that overcome the limitations of traditional solutions.
• SmartCookie introduces a novel split-proxy defense architecture that intelligently divides labor between a high-speed P4 programmable switch (hardware) and a lightweight eBPF component (software).
• The hardware layer (P4 switch) handles high-volume traffic, securely generating SYN cookies with cryptographic hashes and tracking verified connections using approximate data structures like Bloom filters to conserve memory.
• The software layer (eBPF agent on the server) maintains exact connection state for header translations and meticulously handles any false positives from the hardware layer, ensuring accuracy and resource protection.
• SmartCookie achieves superior performance, maintaining near-baseline latency (1.71 ms) for legitimate traffic and zero packet loss up to 136 mpps under extreme attack rates, significantly outperforming prior solutions like Jocken (11.12 ms latency, 36 mpps capacity).
• This layered approach, combining approximate hardware-based filtering with precise software-based verification, offers a scalable, secure, and performant model for mitigating SYN floods, with potential applicability to other network protocols, attacks, and intrusion detection/prevention systems.

About the Speaker(s)

Sophia Yoo is the presenter of the "SmartCookie" research, which she shared at USENIX Security '24. Her work focuses on developing advanced network security defenses, particularly leveraging programmable data planes and innovative architectural approaches to address long-standing challenges like SYN flooding. While specific biographical details such as her title and affiliation were not provided in the talk or metadata, her presentation demonstrated deep expertise in network security, distributed systems, and the practical application of programmable networking technologies.