Your Load Balancer is Your New Perimeter: Attacks & Defenses at Scale

Your Load Balancer is Your New Perimeter: Attacks & Defenses at Scale

Speakers: Arjun Sharma, Lead, Global Load Balancer Service, IBM Cloud

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=6o9PpsDXygs

Overview

In an era where enterprise security perimeters are increasingly complex and costly, a fundamental vulnerability often remains overlooked: the load balancer. Arjun Sharma, who leads IBM Cloud's global load balancer service, delivered a compelling talk at BSides SF, "Your Load Balancer is Your New Perimeter," highlighting how easily sophisticated edge security can be bypassed due to common misconfigurations. Drawing from over a decade of experience building and securing infrastructure that handles billions of requests monthly, Sharma vividly illustrated how attackers exploit these blind spots to gain direct access to backend applications, rendering expensive security investments effectively useless.

The core message of the talk is encapsulated in Sharma's analogy: "It's like buying a really expensive front door for a house which has no back walls." Organizations often invest heavily in enterprise-grade security solutions like CDNs, WAFs, and DDoS protection, assuming these layers shield everything downstream. However, if an attacker can discover the origin IP of a backend application, they can circumvent the entire security stack, directly attacking the vulnerable service. This talk systematically dissects four prevalent attack vectors that lead to such bypasses and, crucially, provides practical, low-cost, and often quick defensive strategies that can be implemented in minutes.

Sharma's presentation serves as a critical wake-up call for security professionals and infrastructure engineers alike. He not only exposes the fragility of perimeter security when origin IPs are compromised but also charts the evolving role of the load balancer as a central security control point, particularly in the emerging AI-native era. By demonstrating real-world attack techniques and offering actionable defensive measures, Sharma empowers attendees with a clear checklist to audit and fortify their infrastructure against these pervasive and often neglected threats.

Background

▶ Watch: The 3 AM call: "Your edge is just a theater" (0:00)

The prevailing assumption in many enterprise architectures is that the edge perimeter—comprising CDNs, WAFs, and DDoS mitigation services—provides an impenetrable shield for all downstream applications. This ideal scenario envisions requests being meticulously scrubbed, sanitized, and authenticated before reaching the backend servers. However, this dream often clashes with a harsh reality: origin IPs frequently leak through misconfigured DNS records and publicly accessible certificate transparency logs. This leakage creates a direct pathway for attackers to bypass the entire edge stack, directly targeting the backend services.

Arjun Sharma emphasized that load balancers, once simple traffic distribution devices, have evolved into one of the most security-critical components in the enterprise stack. This evolution has not always been accompanied by a corresponding shift in security mindset or configuration practices. The paradox is stark: an organization might spend hundreds of thousands of dollars annually on state-of-the-art edge security, but if an attacker can find the true IP address of the origin server, that investment becomes a "security guard in an empty lobby while the attacker is already inside the building" (Sharma, 04:00). Attackers are not "polite" enough to go through the CDN; they actively seek ways around it.

The consequences of such bypasses are severe, potentially leading to data breaches, denial-of-service attacks, and complete system outages, often without the security dashboards reflecting any issues. Sharma noted that while the real data from 2025 on these impacts is significant, the good news is that the defenses against these bypasses are often free and straightforward to implement. The underlying problem stems from a lack of pervasive governance and an over-reliance on the edge to compensate for fundamental misconfigurations deeper within the infrastructure.

Key Findings

▶ Watch: The paradox: Edge protection vs. origin IP leaks (4:00)

Arjun Sharma's talk meticulously detailed four distinct, yet interconnected, attack vectors that allow adversaries to bypass sophisticated edge security and directly target backend applications. These findings underscore the critical importance of treating the load balancer as a primary security perimeter.

1. Origin IP Disclosure via Certificate Transparency (CT) Logs: This is a surprisingly common and often overlooked vulnerability. Every time an SSL/TLS certificate is issued by a Certificate Authority (CA) like Let's Encrypt or Digicert, it is logged into public, immutable, and searchable Certificate Transparency logs. While designed for accountability, these logs become an attacker's goldmine. If a certificate for a subdomain (e.g., admin.example.com) points directly to the load balancer's IP address instead of being proxied through the CDN, the origin IP is publicly exposed. This allows attackers to directly connect to the load balancer, bypassing all WAF, DDoS, and rate-limiting protections provided by the edge.

1. Exploiting Exposed Services on the Load Balancer: Once the load balancer's IP is discovered, the next step for an attacker is to port scan it to identify exposed services. Many organizations mistakenly assume that services behind a firewall are inherently protected. However, if an internal service like Redis (often on port 6379) or an HAProxy stats page (often on port 8080) is bound to a wildcard IP (0.0.0.0) and accessible from the internet, it becomes a critical vulnerability. Attackers can then directly interact with these services, potentially extracting sensitive data like API keys, database passwords, and session tokens from Redis, or mapping the entire internal infrastructure from an HAProxy stats page.

1. Error-Based Enumeration for API Surface Mapping: This subtle yet powerful attack vector exploits inconsistent error responses from applications. While a 404 "Not Found" typically indicates a non-existent path, a 403 "Forbidden" often confirms the existence of an endpoint, merely one that requires authentication or is restricted. Attackers can fuzz common paths and distinguish between 403 and 404 responses to systematically map an application's entire API surface, including administrative endpoints, internal APIs, and debugging tools. This reconnaissance works even through CDNs and WAFs because the error responses originate directly from the application, leaking valuable information about its structure and functionality.

1. DNS "Gray Cloud" Misconfigurations: Many CDN providers, like Cloudflare, offer two primary DNS configurations: Orange Cloud (proxied, providing full security features like WAF, DDoS protection, and rate limiting) and Gray Cloud (unproxied, DNS-only, directly exposing the origin IP). Sharma highlighted that engineering teams sometimes use Gray Cloud configurations for staging, testing, or legacy reasons. If a subdomain configured with a Gray Cloud record points to the same production application as an Orange Cloud record, attackers can simply target the Gray Cloud IP. This completely sidesteps the entire enterprise security stack, making the application susceptible to unmitigated DDoS attacks and allowing direct access without any edge protection.

These findings collectively emphasize that the strength of an organization's security perimeter is ultimately determined by its deepest configuration. The most expensive edge security is rendered moot if fundamental misconfigurations at the load balancer or DNS level provide a direct bypass.

Technical Deep Dive

▶ Watch: Attack 1: Exposing origin IPs via CT logs (5:00)

The technical core of Arjun Sharma's presentation revolves around demonstrating how seemingly minor misconfigurations create critical bypasses for enterprise-grade security. He systematically escalated the attacks, moving from initial reconnaissance to full system compromise.

Attack 1: Origin IP Exposure via Certificate Transparency Logs

The first attack leverages Certificate Transparency (CT) logs, a public, immutable database designed to provide accountability for SSL/TLS certificate issuance (05:00). When a certificate is issued for a domain or subdomain, its details, including the Subject Alternative Name (SAN), are logged. If a subdomain, perhaps used for an internal service or a staging environment, has a certificate that directly resolves to the load balancer's public IP address, that IP becomes publicly discoverable.

Sharma demonstrated this by showing an enterprise setup protected by Cloudflare Enterprise. A curl command to the main domain would correctly return a 403 Forbidden, indicating the WAF and other protections are active (06:00). However, by querying CT logs, an attacker could find a subdomain (e.g., dev.example.com) whose certificate directly reveals the load balancer's IP. A simple dig or nslookup on this subdomain would expose the raw IP address. Subsequently, a curl command directly to this IP would bypass Cloudflare entirely, returning a 200 OK and granting unfiltered access to the load balancer, devoid of any WAF, rate limiting, or DDoS protection headers (08:00).

Attack 2: Exploiting Exposed Services on the Load Balancer

Building on the success of Attack 1, once the load balancer's public IP is known, the attacker proceeds to port scan it. The assumption that services behind a firewall are secure is often flawed if those services are bound to wildcard interfaces (0.0.0.0) and made accessible from the internet. Sharma highlighted common culprits like Redis (default port 6379) and HAProxy stats pages (default port 8080) (11:00).

In the demonstration, an nmap or similar port scan would reveal these open ports. Connecting to Redis via redis-cli often requires no password by default, allowing an attacker to run commands like keys * to dump all cached data. This data frequently includes highly sensitive information such as API keys, database passwords, and user session tokens, all in plain text (12:00). Similarly, accessing an exposed HAProxy stats page provides a comprehensive map of the backend infrastructure, revealing service names, health checks, and traffic patterns—an invaluable resource for an attacker planning further exploitation.

Attack 3: Error-Based Enumeration

This attack vector is more subtle, focusing on information leakage through inconsistent error responses. Most applications return a 403 Forbidden for restricted but existing paths and a 404 Not Found for non-existent paths. From an attacker's perspective, a 403 response is a confirmation of an existing endpoint (15:00).

The attacker would begin by fuzzing common paths using a wordlist against the target application (even through a CDN/WAF, as these errors originate from the backend). By observing the difference between 403 and 404 responses, they can systematically enumerate the application's entire API surface, including hidden administrative interfaces, internal debugging tools, and sensitive API endpoints. Sharma emphasized that "these error messages are documentation for these attackers," allowing them to map in minutes what developers took months to build (16:00).

Attack 4: DNS "Gray Cloud" Misconfigurations

The final attack vector targets DNS misconfigurations, specifically the difference between proxied and unproxied DNS records provided by CDNs like Cloudflare. An Orange Cloud record proxies traffic through the CDN, providing WAF, DDoS, and rate-limiting protections. A Gray Cloud record, however, is DNS-only, directly exposing the origin IP (18:00).

Sharma explained that sometimes, for staging or legacy purposes, a subdomain might be configured with a Gray Cloud record pointing to a production application's IP. An attacker would first curl the main, Orange Cloud-protected domain, receiving a 403 Forbidden due to WAF rules. They would then curl the Gray Cloud subdomain, which, if misconfigured, would return a 200 OK, completely bypassing the CDN's security (20:00). From this point, the attacker can launch unmitigated DDoS attacks against the exposed origin, as there are no rate limits or bot protections. This can lead to a total outage, with security dashboards showing "green" because the attack never hit the monitored edge.

The Load Balancer in the AI-Native Era

Sharma also touched upon the future role of load balancers, especially in the AI-native era. With AI agents increasingly interacting with multiple tools (Slack, GitHub, databases) via JSON RPC and other stateful, long-lived connections with bursty patterns, the load balancer is evolving into an agent gateway (26:00). This new role demands centralized tool registries, session management (tool slicing), aggregated responses, and a shift from simple connection counting to token-weighted consumption for rate limiting to throttle rogue AI agents. This transformation further solidifies the load balancer's position as a critical security and control point, requiring consistent identity, observability, and security posture enforcement.

Demo / Proof of Concept

▶ Watch: Defense 1: Network isolation and origin authentication (8:00)

While the talk was delivered as a presentation rather than a live, interactive demonstration, Arjun Sharma systematically walked the audience through the conceptual steps and expected outputs for each attack vector, effectively serving as a proof of concept for each vulnerability. He clearly outlined the commands and their results, making the attack flow tangible and comprehensible.

For the first attack, Origin IP Exposure via Certificate Transparency Logs, Sharma described starting with a curl command against a main domain, expecting a 403 Forbidden from a protective service like Cloudflare. He then explained how an attacker would query public CT logs to identify a subdomain certificate that reveals the true origin IP. Using dig or nslookup on this identified subdomain, the attacker would resolve the direct IP address. The proof of concept concluded with a curl directly to this origin IP, demonstrating a 200 OK response, signifying a complete bypass of the edge security stack, with no Cloudflare headers or protections evident (06:00-08:00).

The second attack, Exploiting Exposed Services on the Load Balancer, built upon the first. With the load balancer's IP in hand, Sharma detailed how an attacker would perform a port scan (e.g., using nmap) to identify open ports like 6379 (Redis) or 8080 (HAProxy stats). He then described connecting to the exposed Redis instance using redis-cli, highlighting that often no password is required. The proof of concept involved executing keys * to reveal cached API keys, database passwords, and session tokens—all in plain text. Similarly, accessing the HAProxy stats page would expose a detailed map of the internal infrastructure, including backend servers and their states (11:00-12:00).

For Error-Based Enumeration, Sharma outlined the process of probing different endpoints on the application. After an nslookup on a leaked subdomain to get the origin, an attacker would curl various paths. The core of the proof of concept was observing the difference between a 404 Not Found (non-existent path) and a 403 Forbidden (existing but restricted path). This distinction, when combined with a wordlist and fuzzing, allows an attacker to rapidly map out the application's entire API surface, including sensitive administrative or internal endpoints (15:00-16:00).

Finally, for the DNS "Gray Cloud" Misconfigurations, Sharma demonstrated the bypass by first curling the main, Orange Cloud-protected domain, which would correctly return a 403 Forbidden due to WAF rules. He then explained how a leaked subdomain configured with a Gray Cloud (unproxied DNS) record, also pointing to the production application, would be targeted. A curl to this Gray Cloud subdomain would yield a 200 OK, illustrating the complete circumvention of the CDN's protective layers. The final step in this proof of concept involved attempting rate-limited requests against the Orange Cloud (resulting in a 429 Too Many Requests) versus the Gray Cloud (showing no limits, demonstrating susceptibility to unmitigated DDoS attacks) (19:00-21:00).

Across all these scenarios, Sharma's narrative served as a clear, step-by-step technical walkthrough, effectively demonstrating the attack vectors and their immediate impact, even without live, interactive screen sharing.

Defensive Implications

▶ Watch: Attack 2: Port scanning exposed load balancer IPs (10:00)

Arjun Sharma stressed that the defenses against these bypasses are often simple, cost nothing, and can be implemented quickly. The overarching principle is to adopt a defense-in-depth strategy, assuming that the network will be compromised, and enforcing security at every layer.

Defending Against Origin IP Disclosure (Attack 1)

The primary defense against origin IP leakage involves a two-pronged approach:

1. Network Isolation: The most critical step is to configure security group rules or network ACLs to only allow inbound traffic to your application from your load balancer's specific IP addresses or subnets (08:00). This means no any/any rules should permit internet traffic directly to your backend.

• 5-minute fix: Audit existing security group rules. Identify and remove any 0.0.0.0/0 (any/any) ingress rules for your application's ports. Add a new, highly specific rule allowing ingress only from your load balancer's IP(s). Crucially, test thoroughly to ensure production workloads are unaffected before deleting the broad rules.

1. Origin Authentication on the Load Balancer: The load balancer should be configured to validate incoming requests, ensuring they truly originate from the CDN. This can be achieved by:

• Validating CDN-specific headers: Many CDNs set unique headers like X-Origin-Verify. The load balancer should inspect and validate these.
• Mutual TLS (mTLS): Enforce mTLS between your CDN edge and your load balancer. This requires both the client (CDN) and server (load balancer) to present and validate certificates, ensuring that even if an attacker discovers the origin IP, they cannot establish a connection without the CDN's client certificate (08:00, 31:00).

Defending Against Exposed Services (Attack 2)

Once the load balancer's IP is exposed, internal services must be hardened:

1. Bind Services to Localhost/Internal Interfaces: Crucially, internal services like Redis or HAProxy stats should never be bound to 0.0.0.0 (wildcard), which makes them accessible from any network interface. Instead, bind them to 127.0.0.1 (localhost) or a specific internal network interface that is not publicly routable (13:00).

• 10-minute fix: Use ss -tnlp to identify all services bound to TCP sockets and look for 0.0.0.0 bindings. For services like Redis, this can be a one-line configuration change (e.g., bind 127.0.0.1). Restart services after reconfiguration.

1. Enforce Authentication and TLS: For any service that must be accessible internally, enforce strong authentication (e.g., Redis requirepass with a strong, generated password using openssl rand -hex 32). Additionally, enable TLS for all internal communication to encrypt data in transit, even within what is assumed to be a trusted network (13:00).

Defending Against Error-Based Enumeration (Attack 3)

The key here is to eliminate information leakage through inconsistent error messages:

1. Uniform Error Responses: Implement a policy where all unauthenticated requests receive a generic 404 Not Found response, regardless of whether the path actually exists or requires authentication (17:00). This removes the distinction between 403 and 404 that attackers rely on.

• Quick win: Implement a default deny policy at the root location of your application or load balancer configuration. Return a 404 for every request by default, and explicitly define location blocks for paths that are genuinely public and require specific responses.

Defending Against DNS "Gray Cloud" Misconfigurations (Attack 4)

This attack highlights the need for robust DNS hygiene:

1. Gold Standard: VPC with Private DNS: For enterprise-grade deployments, the ideal solution is to deploy your origin applications within a Virtual Private Cloud (VPC) with private DNS. Your origin servers should have no public IP addresses and reside in private subnets. DNS resolution for the origin should only occur within the private VPC, and traffic from the CDN should route through a private path directly into the VPC. This completely eliminates the possibility of Gray Cloud records, as there's no public IP to point them to (21:00).

1. Alternative (if Public IP is Necessary): If public IPs are unavoidable (e.g., for legacy reasons or migrations):

• CDN IP Whitelisting: Configure your security groups to only allow inbound traffic from your CDN's known, published IP ranges (e.g., Cloudflare's IP list).
• mTLS and Origin Pull Header Validation: As with Attack 1, enforce mTLS between the CDN and load balancer. Additionally, validate origin pull headers that your CDN sets (e.g., CF-Connecting-IP, X-Forwarded-For). Reject any request that comes directly to the IP address without the expected host name or CDN headers (22:00).
• 10-minute fix: Actively scan all your DNS records for Gray Cloud configurations. Toggle any Gray Cloud records pointing to production applications to Orange Cloud (proxied). Delete any residual staging or testing configurations. Verify with dig that no domain returns your original IP without passing through the CDN (23:00).

Comprehensive Rate Limiting and AI Era Defenses

Sharma also advocated for a multi-layered rate-limiting strategy:

• Edge: Handles noisy, volumetric attacks (IP-based limits, bot detection, target ~1000 requests/minute).
• Load Balancer: Catches distributed attacks (concurrent connections, header validation, target ~500 requests/minute).
• Application: Enforces identity-based, token-weighted quotas per authenticated user (target ~100 requests/user).
• Backend Services: Implement circuit breakers and backpressure mechanisms for graceful degradation (24:00).

In the AI-native era, load balancers will evolve into agent gateways, requiring new defenses like centralized tool registries, tool slicing for session management, and token-weighted consumption for throttling rogue AI agents. This new perimeter demands consistent identity, security posture, and observability (OpenTelemetry integration) (26:00-28:00).

Organizational Best Practices

Beyond technical fixes, Sharma emphasized organizational measures:

• Cloud Policies: Implement cloud policies to prevent bad configurations at an organizational level.
• Secure Terraform Templates: Provide developers with secure, pre-configured infrastructure-as-code templates.
• Automated Scanning: Automate scanning for misconfigurations, as "someone will always go rogue" (29:00).

The core message for defenders is clear: treat your load balancer as a security tool, not just a router. Implement defense in layers, combine network isolation with service hardening and proper DNS hygiene at scale, and automate governance to ensure pervasive security.

Key Takeaways

• Load Balancer as the New Perimeter: The load balancer is no longer just a traffic router but a critical security control point. It must be treated as a primary security tool, overseeing everything from traffic distribution to authentication and threat mitigation.
• Misconfigurations Undermine Expensive Security: Even with significant investments in enterprise-grade CDNs, WAFs, and DDoS protection, simple misconfigurations in DNS, certificate logs, or backend service bindings can create direct bypasses, rendering the entire edge security stack ineffective.
• Defense in Layers is Paramount: A robust security posture requires a multi-layered approach, combining network isolation (restricting inbound traffic to the load balancer only), service hardening (binding services to localhost, enforcing authentication and TLS), and rigorous DNS hygiene.
• Information Leakage Fuels Attacks: Subtle cues like inconsistent error messages (e.g., distinguishing 403 from 404) provide attackers with valuable information to map API surfaces. Standardizing error responses to a generic 404 for unauthenticated requests is a quick and effective defense.
• DNS Hygiene is Non-Negotiable: "Gray Cloud" or unproxied DNS records pointing directly to origin IPs are critical vulnerabilities. The gold standard is a VPC with private DNS; otherwise, strict IP whitelisting, mTLS, and origin header validation are essential. Regularly audit and correct DNS records.
• Evolving Role in the AI-Native Era: As AI agents proliferate and utilize stateful, bursty connections, the load balancer's role will expand to an "agent gateway," requiring advanced capabilities like token-weighted rate limiting, centralized tool registries, and identity enforcement for AI interactions.

About the Speaker(s)

Arjun Sharma is a distinguished expert in infrastructure security, currently leading IBM Cloud's global load balancer service. With 13 years of extensive experience, he has been instrumental in building and securing infrastructure that processes billions of requests each month for enterprise customers worldwide. His daily responsibilities include conducting architectural reviews, collaborating with security teams, ensuring robust security controls, performing threat modeling, and critically, preventing infrastructure bypasses. Sharma's work focuses on the practical aspects of securing large-scale cloud services, making him a highly credible voice on the evolving challenges and solutions in perimeter security.