A Large-Scale Measurement Study of the PROXY Protocol and its Security Implications

A Large-Scale Measurement Study of the PROXY Protocol and its Security Implications

Speakers: Stijn Pletinckx, PhD student, UC Santa Barbara

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=voirgWowHK8

Overview

In a critical presentation at the NDSS Symposium, Stijn Pletinckx from UC Santa Barbara unveiled a comprehensive large-scale measurement study on the PROXY protocol, revealing widespread misconfigurations and significant security vulnerabilities. This talk sheds light on how a protocol designed to enhance visibility in load-balanced environments can be weaponized by attackers to bypass security controls, access sensitive internal infrastructure, and even turn email servers into persistent open relays. The research underscores a fundamental flaw in the common deployment assumptions of the PROXY protocol, where backend servers often blindly trust client information without adequate validation.

The PROXY protocol, operating at Layer 4, serves a crucial role in modern network architectures, particularly those employing load balancers or reverse proxies. Its primary function is to convey original client connection information—such as IP address and port—from the proxy server to the backend server. This allows backend systems, which would otherwise only see the proxy's IP, to log accurate client data and apply IP-based access controls. However, Pletinckx's study demonstrates that despite the perceived security benefits of a proxy setup (e.g., access control, DoS protection, central blocklisting), an adversary can directly inject a PROXY header into their connection, effectively bypassing the proxy and directly interacting with the backend.

The implications of this research are profound, highlighting a critical gap in network security practices. The study meticulously details two primary attack vectors: a regular bypass, where an attacker directly connects to a backend server by injecting a PROXY header, and a more insidious spoofed bypass, where the attacker also spoofs the client IP address within that header. These bypasses undermine IP-based access controls and expose internal services, sensitive data, and vulnerable systems that were thought to be protected. The findings call for immediate action from system administrators and software vendors to re-evaluate and secure their PROXY protocol implementations.

Background

▶ Watch: Introduction to PROXY protocol and its core function (0:00)

Modern web and application architectures frequently employ load balancers and reverse proxies to distribute client requests across multiple backend servers, enhance performance, and provide a layer of security. In such a setup, a client initiates a connection to the proxy server, which then selects an appropriate backend server to forward the request. A fundamental challenge arises from this architecture: when the backend server processes the request, its logs typically record the IP address of the proxy server, not the original client. This obscures crucial client information, making it difficult for backend applications to implement IP-based access controls, perform accurate logging, or analyze user behavior effectively.

To address this information asymmetry, various protocols and mechanisms have emerged. On Layer 7 (HTTP), the X-Forwarded-For header is a common solution, allowing the proxy to append the client's IP address to the HTTP request. However, for Layer 4 protocols (TCP/UDP) where such application-level headers are not natively present, the PROXY protocol was developed. The PROXY protocol works by prepending a small, human-readable header to the beginning of the connection between the proxy server and the backend server. This header contains essential client information, including the client's source IP address and port, and the destination IP address and port that the proxy originally connected to. This allows the backend server to "see" the original client's details, enabling more accurate logging and the application of client-specific policies.

The perceived security model associated with a proxy setup assumes that the proxy acts as a gatekeeper. Security measures such as access control lists, DDoS protection, and central blocklisting are typically implemented at the proxy level. It is commonly assumed that if an attacker is blocked by the proxy, they cannot reach the backend. Furthermore, if an attacker were to attempt a direct connection to the backend, it is often believed that the backend, expecting a PROXY header, would reject the connection if the header is absent or malformed. This expectation forms the basis of many security designs.

However, the core flaw identified by Pletinckx's research is that the PROXY protocol itself lacks any inherent authentication or validation mechanism to ensure that the header originates from a legitimate, trusted proxy server. An attacker can simply craft a connection that includes a valid PROXY header and send it directly to the backend server, entirely bypassing the front-end proxy. This direct injection circumvents any security controls enforced by the proxy. The protocol specification places the burden of trust entirely on the system administrator, requiring proactive measures such as maintaining a trusted list of proxy server IP addresses (e.g., via firewall rules) or performing connection verification based on the source IP of the incoming connection. Without these explicit measures, backend servers are vulnerable.

This vulnerability is exacerbated by the ability to spoof the client IP address within the injected PROXY header. If the backend server relies on the IP address provided in the header for access control, an attacker can craft a header with an arbitrary IP address (e.g., a local network IP or localhost) to bypass IP-based restrictions. The study categorizes these two attack types as regular bypass (bypassing the proxy by injecting a valid header with the attacker's own IP) and spoofed bypass (bypassing IP-based controls by injecting a valid header with a fabricated client IP). The existence of these bypasses fundamentally undermines the security assumptions of many PROXY protocol deployments.

Key Findings

▶ Watch: Spoofing client IP addresses to bypass controls (2:40)

The research embarked on a large-scale measurement study across the entire IPv4 address space to quantify the prevalence of PROXY protocol bypasses and assess their real-world security implications. The methodology involved scanning HTTP, SMTP, and SSH services. The team utilized ZMAP for initial port scanning and a modified version of ZGRAP to complete Layer 7 connections, specifically tailored to inject PROXY headers during the connection setup.

The scanning pipeline was meticulously designed to differentiate between various responses and confirm bypasses:

1. Regular Probe: A standard connection without a PROXY header. If successful, the target was discarded as there was nothing to bypass.
2. Injected Proxy Probe (Own IP): If the regular probe failed, a PROXY header containing the scanner's own IP address was injected.
3. Malformed Proxy Probe: To confirm that the PROXY header was indeed the cause of a successful response, a malformed header (e.g., protocol "ABC") was sent. If the malformed probe failed while the valid injected probe succeeded, it confirmed a regular bypass. If both succeeded, it was considered a false positive, indicating the backend might be tolerant of arbitrary prefixes.
4. Spoofed Proxy Probe: If the initial injected probe (with the scanner's own IP) failed, attempts were made with spoofed IP addresses. Specifically, the localhost address (127.0.0.1) and three common internal infrastructure prefixes (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) were used. The malformed probe confirmation step was repeated to verify a spoofed bypass.

The prevalence results were striking:

• On HTTP, over 170,000 hosts were found to happily accept injected PROXY headers.
• The numbers were significantly higher for other protocols, with 2.3 million hosts positively reacting to injected PROXY headers on both SMTP and SSH.

The security implications of these bypasses were rigorously assessed:

HTTP Vulnerabilities:

• By classifying the web pages returned, the study found that for the experiment group (with injected headers), over 60% of responses consisted of boilerplate code. This suggests that system administrators might have been experimenting with the protocol but failed to properly configure or secure these instances, leaving them without meaningful content.
• More critically, a deeper manual analysis of 200 sampled web pages from the "miscellaneous" category revealed that 36% hosted some form of sensitive content or provided access to vulnerable systems. Examples included:
• Home automation systems, granting control over lights or window blinds.
• Monitoring services, exposing telemetry data from data centers.
• Extrapolating this to the entire "miscellaneous" group suggests potentially more than 4,000 pages with sensitive content could be accessed by simply injecting a PROXY header.
• For spoofed bypasses on HTTP, a slight increase in success rate was observed, particularly when spoofing IPs within the 10.0.0.0/8 and 172.16.0.0/12 prefixes.

SMTP Vulnerabilities (Open Relays):

• The most severe finding was the ability to turn SMTP servers into open relays using PROXY header injection, specifically with spoofed localhost IPs. Many email server software, such as Postfix, by default, relay emails originating from the localhost address without authentication, assuming internal authenticity.
• By spoofing the localhost IP (127.0.0.1) in the PROXY header, an attacker can convince the backend email server that an email originated internally, allowing them to send emails in the name of any email address without authentication.
• The study identified over 370 email servers vulnerable to this specific attack. This number is conservative, as it only tested with the localhost address; an adversary with more infrastructure knowledge could target specific internal IP ranges.
• A crucial observation was the persistence of these open relays. Traditional open relay scanners do not incorporate PROXY header injection, thus failing to detect these vulnerabilities. Consequently, the identified vulnerable servers remained online for days and weeks after the experiment, undetected by conventional scanning tools.

In conclusion, the key findings demonstrate that the PROXY protocol, when deployed without proper validation, is a significant attack vector. It enables attackers to bypass assumed security perimeters, access sensitive information, control internal systems, and leverage email infrastructure for malicious purposes, all while often remaining undetected by existing security measures.

Technical Deep Dive

▶ Watch: Large-scale measurement study methodology explained (3:30)

The PROXY protocol's simplicity is both its strength and its Achilles' heel. It operates by prepending a single line of text, known as the PROXY header, to the beginning of a connection. This header provides the necessary context for the backend server to identify the original client. The general format of the PROXY header is:

PROXY {PROTOCOL} {CLIENT_IP} {PROXY_IP} {CLIENT_PORT} {PROXY_PORT}\r\n

Let's break down the components:

• PROXY: This is the fixed proxy preamble that signifies the start of a PROXY protocol header.
• {PROTOCOL}: Specifies the network protocol being used, typically TCP4 for IPv4 TCP connections, TCP6 for IPv6 TCP, or UNKNOWN if the protocol is not recognized.
• {CLIENT_IP}: The IP address of the original client. This is the critical piece of information conveyed.
• {PROXY_IP}: The IP address of the proxy server.
• {CLIENT_PORT}: The ephemeral port used by the client.
• {PROXY_PORT}: The port on the proxy server that received the client's connection.
• \r\n: The standard CRLF (carriage return, line feed) sequence, marking the end of the header.

The core vulnerability stems from the fact that a backend server configured to expect and parse this header typically does so without verifying the source of the connection. An attacker can craft a malicious PROXY header, append their intended Layer 7 payload (e.g., an HTTP GET request or an SMTP command), and send it directly to the backend server. The backend, upon receiving the PROXY preamble, will parse the subsequent fields, effectively treating the attacker's connection as if it originated from a legitimate proxy.

The scanning pipeline developed for this study was instrumental in identifying these vulnerabilities at scale:

1. Initial Regular Probe (No Header):

• Purpose: Identify hosts that are not protected by a proxy or do not require a PROXY header for access.
• Action: Send a standard Layer 7 request (e.g., GET / HTTP/1.0 for HTTP, EHLO example.com for SMTP, SSH-2.0-OpenSSH_... for SSH) directly to the target.
• Outcome: If a successful response is received, the target is excluded from further testing, as no bypass is necessary.

1. Injected Proxy Probe (Attacker's IP):

• Purpose: Test for a regular bypass.
• Pre-condition: Initial regular probe failed (e.g., connection timed out, rejected, or received a non-successful status code).
• Action: Construct a PROXY header using the scanner's own public IP address as both {CLIENT_IP} and {PROXY_IP} (or using 127.0.0.1 for {PROXY_IP} if the backend expects a local proxy), followed by the standard Layer 7 request. Example: PROXY TCP4 198.51.100.1 198.51.100.1 12345 80\r\nGET / HTTP/1.0\r\n\r\n.
• Outcome: If a successful Layer 7 response is now received, it indicates a potential bypass.

1. Malformed Proxy Probe (Confirmation):

• Purpose: Differentiate between a genuine PROXY protocol bypass and a backend that simply tolerates arbitrary prefixes or garbage data.
• Pre-condition: The Injected Proxy Probe (Attacker's IP) was successful.
• Action: Send a PROXY header with a syntactically incorrect field, specifically modifying the protocol field to an invalid value, e.g., PROXY ABC ....
• Outcome:
• If the malformed probe fails (e.g., connection reset, HTTP 400 Bad Request, no resource provided), it strongly confirms that the correct PROXY header was indeed responsible for the previous success, thus classifying it as a regular bypass.
• If the malformed probe also succeeds, it suggests a false positive where the backend is not strictly enforcing the PROXY protocol format but rather just accepting any initial data.

1. Spoofed Proxy Probe (Internal IPs):

• Purpose: Test for a spoofed bypass, specifically targeting IP-based access controls.
• Pre-condition: The Injected Proxy Probe (Attacker's IP) failed. This implies the backend might be performing some IP-based filtering, or perhaps only accepting connections from specific internal IPs as proxies.
• Action: Construct PROXY headers where the {CLIENT_IP} field is set to commonly used internal IP addresses: 127.0.0.1 (localhost), 10.0.0.1 (example from 10.0.0.0/8), 172.16.0.1 (example from 172.16.0.0/12), and 192.168.0.1 (example from 192.168.0.0/16). The {PROXY_IP} can remain the scanner's actual IP or be set to 127.0.0.1 depending on the target's likely configuration. The Layer 7 request follows.
• Outcome: If a successful Layer 7 response is received, followed by a failure on the corresponding malformed probe, it confirms a spoofed bypass.

A particularly potent example of a spoofed bypass is the SMTP open relay vulnerability. Many default configurations of mail transfer agents (MTAs) like Postfix are designed to relay emails without authentication if they appear to originate from the localhost address (127.0.0.1). This is a security feature, assuming that any email from localhost has already passed internal authentication or is from a trusted local service. An attacker, by crafting a PROXY header with 127.0.0.1 as the {CLIENT_IP}, can deceive the backend SMTP server into believing the connection originates from localhost. The attacker then sends an email with a spoofed sender address (e.g., FROM: [email protected]). The backend server, trusting the localhost origin, happily processes and relays this email without requiring any credentials, effectively turning the server into an unauthenticated open relay. The persistence of these open relays is a critical finding because current vulnerability scanners, lacking PROXY protocol awareness, cannot detect them, allowing them to operate unnoticed for extended periods.

The ethical considerations of this research were carefully managed. All scans included opt-out mechanisms, and spoofed HTTP requests used HEAD requests only to prevent accidental data leakage. All emails sent during the SMTP experiments originated from and were delivered to addresses under the researchers' control. Furthermore, all affected parties were notified through responsible disclosure, leading to some bounty awards.

Demo / Proof of Concept

▶ Watch: Prevalence: Millions of hosts vulnerable to bypass (5:00)

While the talk did not feature a live, interactive demonstration in the traditional sense, the entire measurement study itself served as a large-scale, distributed proof of concept. The methodology and findings directly demonstrated the feasibility and impact of PROXY protocol bypasses across millions of internet-facing systems. The "demo" was implicitly provided by the sheer volume and nature of the vulnerabilities discovered.

Specifically, the discovery of home automation systems and monitoring services that became accessible by simply injecting a PROXY header with the scanner's own IP address stands as a powerful proof of concept for the regular bypass. The research team could gain unauthorized access to controls for lights, window blinds, and sensitive telemetry data, clearly illustrating how systems presumed to be protected by a proxy were directly exposed. This wasn't a simulated environment; these were real-world systems found on the internet.

For the spoofed bypass, the most compelling proof of concept was the successful transformation of over 370 SMTP servers into open relays. The ability to send emails in the name of any address (e.g., a CEO) without authentication, simply by spoofing 127.0.0.1 in the PROXY header, directly proves the efficacy and severe implications of this attack vector. The fact that these servers remained persistent open relays, undetected by conventional scanning tools, further underscores the practical impact and stealth of this vulnerability. These weren't theoretical attacks; they were confirmed, exploitable conditions found in production environments. The study’s meticulous process of verification, including the use of malformed headers to confirm the PROXY protocol’s role, robustly validates these findings as practical proofs of concept.

Defensive Implications

▶ Watch: Security implications: Boilerplate code on vulnerable hosts (5:30)

The findings of this study necessitate immediate and decisive action from system administrators, network architects, and software developers to secure deployments utilizing the PROXY protocol. The fundamental defensive principle is to never implicitly trust client information provided via the PROXY header without explicit validation of the source.

Here are the key defensive implications and recommended actions:

1. Implement Strict Allow Lists (Whitelisting) for Proxy Sources: This is the single most critical defense. Backend servers configured to parse the PROXY protocol header must maintain an explicit list of trusted IP addresses from which these headers are accepted. Any connection originating from an IP address not on this allow list that attempts to send a PROXY header should be immediately rejected or treated as a regular, un-proxied connection (ignoring the header). This prevents both regular and spoofed bypasses by ensuring only legitimate proxy servers can supply client information.

• Action for Admins: Configure firewalls (e.g., iptables, firewalld, cloud security groups) to only allow inbound connections to backend servers on PROXY-enabled ports from the IP addresses of your trusted load balancers or proxy servers. Additionally, if the backend application or web server software supports it (e.g., Nginx, Apache, Postfix), configure it to explicitly trust PROXY headers only from a defined list of source IPs.
• Action for Software Vendors: The talk highlights that popular backend software like Nginx or Apache, while supporting the PROXY protocol, often lacks a built-in, default-deny mechanism for PROXY header acceptance. Unlike X-Forwarded-For which often requires explicit configuration to trust, the PROXY protocol often defaults to trusting any source. Vendors should implement a default-deny policy for PROXY headers, requiring administrators to explicitly define trusted sources, similar to how X-Forwarded-For is handled.

1. Network Segmentation and Least Privilege: Backend servers should be deployed in highly segmented network zones, ideally not directly exposed to the public internet. Access to these backend segments should be strictly controlled, allowing connections only from the designated proxy tier. This physical or logical isolation adds a layer of defense, ensuring that even if a configuration error exists, direct connections are difficult to establish.

1. Regular Auditing and Configuration Review: System administrators should regularly audit their backend server configurations to ensure that PROXY protocol settings are correctly applied and that allow lists are up-to-date. This includes reviewing firewall rules, server software configurations (e.g., nginx.conf, apache.conf, main.cf for Postfix), and cloud security group policies.

1. Enhance Vulnerability Scanning: Current vulnerability scanners, particularly those for open relays, need to be updated to incorporate PROXY header injection techniques. Organizations should ensure their internal security assessments and external penetration tests include checks for these PROXY protocol bypasses. This will help detect persistent vulnerabilities that are currently missed.

1. Educate and Train: Developers and system administrators need to be educated on the security implications of the PROXY protocol. Awareness of the protocol's lack of inherent authentication and the necessity of proactive validation measures is crucial for secure deployment.

1. Consider Alternatives or Enhancements for Sensitive Systems: For extremely sensitive backend systems, consider additional layers of security beyond just IP-based trust. This could include mutual TLS authentication between the proxy and the backend, or other forms of cryptographic authentication for the PROXY header itself (though this would require protocol extensions).

By implementing these defensive measures, organizations can close the critical security gap identified by Pletinckx's research, ensuring that the PROXY protocol serves its intended purpose without introducing severe bypass vulnerabilities.

Key Takeaways

• The PROXY protocol is widely adopted but frequently misconfigured, leading to severe security vulnerabilities across millions of internet-facing hosts.
• Attackers can perform a regular bypass by directly injecting a valid PROXY header into a connection, circumventing the front-end proxy's security controls.
• A more advanced spoofed bypass allows attackers to fabricate the client IP address within the PROXY header, enabling them to bypass IP-based access controls on backend systems.
• Real-world impact includes unauthorized access to sensitive internal infrastructure (e.g., home automation, monitoring systems) and the exposure of private data.
• Over 370 SMTP servers were found to be vulnerable to becoming persistent open relays by spoofing the localhost IP via the PROXY protocol, allowing unauthenticated email sending.
• The primary defense mechanism is for backend servers to implement strict allow lists (whitelists) of trusted proxy IP addresses from which PROXY headers will be accepted, rejecting or ignoring headers from unknown sources. Current backend software often lacks this crucial default-deny security posture.

About the Speaker(s)

Stijn Pletinckx is a PhD student at UC Santa Barbara. His research interests include network security and measurement studies, focusing on understanding the prevalence and implications of protocol misconfigurations and vulnerabilities in real-world internet deployments. This talk at the NDSS Symposium represents a significant contribution to the field, highlighting critical, previously unaddressed security concerns related to widely used network protocols.