Expanding eBPF’s Reach: From Batteries-Included Auto-Instrumentation To E2E Observab... Dom Del Nano

Expanding eBPF’s Reach: From Batteries-Included Auto-Instrumentation To E2E Observability Pipelines

Speakers: Dom Del Nano, CEO and Founder, Cosmic; CNCF Pixie core maintainer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=I236sjooftw

Overview

In this insightful KubeCon EU talk, Dom Del Nano, CEO and founder of Cosmic and a CNCF Pixie core maintainer, delves into the evolving landscape of observability in modern cloud-native environments. The presentation, titled "Expanding eBPF’s Reach: From Batteries-Included Auto-Instrumentation To E2E Observability Pipelines," highlights the transformative potential of eBPF (extended Berkeley Packet Filter) in overcoming the inherent complexities of monitoring highly distributed, polyglot microservice architectures. Del Nano argues that while eBPF provides unparalleled visibility, its true power is unlocked when tools move beyond mere data collection to offer actionable insights through sophisticated enrichment, flexible APIs, and domain-specific programmability.

The core message of the talk resonates deeply with the challenges faced by developers and operators today. As monolithic applications have given way to intricate networks of microservices, containers, and Kubernetes, traditional manual instrumentation has become prohibitively expensive and incomplete. eBPF emerges as a kernel-level solution for automatic, system-wide telemetry. However, Del Nano emphasizes that raw eBPF data can be an overwhelming "firehose" of information. The talk meticulously unpacks how advanced eBPF-based tools can tame this data deluge, providing the context and intelligence necessary to debug, secure, and optimize complex systems effectively.

Background

▶ Watch: From Monoliths to Microservices: The Rise of Observability (0:50)

The journey to modern observability began with a stark contrast to today's cloud-native landscapes. In the era of monolithic applications, debugging was relatively straightforward. Issues were contained within a single, large codebase, and the "three pillars of observability" (logs, metrics, traces) were not the critical operational tools they are now. However, the advent of microservices, containers, and Kubernetes fundamentally reshaped application architectures. Today's environments are characterized by services written in diverse programming languages, reliance on SaaS services and third-party APIs, and a variety of data stores. This fragmentation made traditional monitoring approaches, particularly manual instrumentation, incredibly challenging and costly. Instrumenting each component in a polyglot environment required significant effort, often resulting in sparse coverage, inconsistency, and infeasibility for third-party or SaaS components.

This is where eBPF enters the picture as a game-changer for auto-telemetry. By hooking directly into the Linux kernel, eBPF offers a system-wide view, capturing events and data without requiring application code changes or restarts. This kernel-level access enables broad coverage across all languages and frameworks, even allowing introspection into encrypted connections by hooking into TLS libraries to reveal plaintext data. The effort required for instrumentation dramatically decreases, as eBPF programs are written once and provide system-wide data. While manual instrumentation offers high flexibility due to direct code control, eBPF's auto-instrumentation means flexibility becomes tooling-dependent, a crucial factor Del Nano explores.

However, the sheer volume of data generated by system-wide eBPF telemetry presents its own challenge. As Del Nano points out, "observability data, which is notoriously high volume, low value, often becomes a little bit more painful because you now all of a sudden have even more data." This sentiment echoes a post by Bill Mulligan, a community manager for the Cilium umbrella projects, on making eBPF impactful. The core insight is that users don't just want data; they want insights and actionable signals. Therefore, impactful eBPF tools must go beyond mere data collection to provide aggregation, analysis, and the ability to act on the derived information, ensuring that important system signals are not obscured by the data firehose.

Key Findings

▶ Watch: eBPF Data Overload: The Need for Insights, Not Just Data (4:50)

The central finding of Dom Del Nano's talk is that the true power of eBPF in modern observability lies not in its raw data collection capabilities, but in how effectively tools process, enrich, and present that data. To transform the "firehose" of low-level eBPF events into meaningful insights, Del Nano identifies three critical properties that impactful eBPF tools must possess: data enrichment, structured APIs, and programmability.

Firstly, data enrichment is paramount because raw eBPF data, such as network traffic as "bytes on the wire," lacks the necessary context for debugging microservice environments. Tools must enrich this low-level data with higher-level information, such as Kubernetes primitives (pod names, namespaces) and container details, to make it actionable for engineers.

Secondly, structured APIs are essential for flexibility. Since auto-telemetry is performed on the user's behalf, tools need to provide mechanisms to extract, filter, and manipulate the data programmatically. This allows users to define custom views, integrate with other systems, and apply their specific debugging or security scopes, rather than being confined to predefined dashboards.

Finally, programmability enables domain-specific extensibility and processing. This goes beyond simple filtering, allowing users to apply specific observability or security logic directly within the eBPF tool's framework. This capability ensures that the tool can adapt to diverse operational needs and provide highly targeted insights relevant to the specific problem domain, whether it's performance debugging or runtime security.

Del Nano demonstrates these findings by evaluating four prominent eBPF projects—Inspector Gadget, Pulsar, Pixie, and Hubble (from the Cilium project)—against these three properties. While all four projects generally excel in data enrichment and structured APIs, Pixie and Pulsar are highlighted for their advanced programmability, showcasing how they enable users to define complex security policies or perform intricate observability analyses, thus moving from "batteries-included auto-instrumentation" to comprehensive end-to-end observability pipelines.

Technical Deep Dive

▶ Watch: Key Property 1: Data Enrichment for Contextual Insights (6:10)

The technical core of Del Nano's talk meticulously unpacks the three crucial properties that elevate eBPF tools from raw data collectors to powerful insight generators: data enrichment, structured APIs, and programmability. He illustrates these concepts with concrete examples from prominent eBPF projects.

Data Enrichment

eBPF operates at the kernel level, providing low-level data that, while comprehensive, often lacks immediate application context. For instance, tracing network traffic might yield bytes on the wire, but this is less useful than knowing which Kubernetes pod initiated the connection or which service it targeted. Effective eBPF tools must bridge this gap.

• Inspector Gadget provides automatic enrichment by integrating Kubernetes and container primitives directly into its eBPF programs. Del Nano shows a snippet where define and include statements within an Inspector Gadget script automatically populate low-level kernel information with rich application-level context, such as Kubernetes namespace and Kubernetes pod name, as seen in the output of the trace_dns gadget. This makes it effortless to correlate DNS queries with the specific workloads performing them.
• Pixie achieves data enrichment through its Python-like query language, Pixel. When querying DNS events, Pixel allows users to access the pod associated with each event and perform IP to podname lookups on remote addresses. This transforms a raw IP address into an understandable service or pod name, providing a full picture of network interactions, even for external connections. For example, a DNS query output in Pixie would clearly show the pod and namespace for both source and destination (if internal), or resolve external IPs to hostnames.

Structured APIs

To ensure flexibility and allow users to interact with and process the vast amounts of auto-telemetry data, eBPF tools need well-defined, structured APIs.

• Hubble, the observability component of the Cilium project, provides a robust structured API through its Hubble Relay component. This relay exposes network flow data via a gRPC API. The Hubble UI and CLI are built directly on top of this API, demonstrating a "dogfooding" approach. This allows users to build custom integrations or alternative UIs if the standard offerings don't perfectly fit their needs, enabling high-level programmatic access to network flow information.
• Pixie leverages its Pixel language as its primary structured API. Pixel offers Python/Pandas-like data flow programming, making it intuitive for data scientists and engineers to query, filter, and transform observability data. Similar to Hubble, Pixie's UI is entirely powered by Pixel, meaning users can directly modify Pixel code to change visualizations. Client libraries are also provided, allowing third-party applications to interact with Pixie's data processing engine using Pixel scripts as input, yielding metrics, spans, profiles, and events.

Programmability (Domain-Specific Extensibility and Processing)

Programmability is arguably the most advanced property, enabling tools to move beyond generic data access to offer domain-specific logic and processing.

• Pulsar, a runtime security tool, exemplifies programmability in the security domain. It provides a rich interface for defining security policies and rules. Del Nano presents examples of rules, such as flagging sensitive file access to /etc/shadow by any process other than sshd, or detecting suspicious activity like using telnet or netcat. This framework allows security engineers to distill complex security principles into actionable rules, with Pulsar handling the underlying eBPF event capture and rule evaluation, abstracting away the low-level eBPF complexity from the end-user.
• Pixie demonstrates observability-focused programmability with its Pixel language. Beyond basic queries, Pixel includes functions like ns_lookup, which can resolve IP addresses to domain names. Del Nano illustrates this by showing a Pixel script detecting outbound connections. If a destination IP is an internet address, ns_lookup can convert it into a hostname (e.g., resolving an IP to stripe.com), providing immediate context about external API calls. This programmatic functionality is extremely powerful for debugging systems and understanding their external dependencies and behavior from an observability standpoint. This ability to extend and process data within the tool's framework makes eBPF truly powerful for domain-specific analysis.

These detailed examples underscore Del Nano's argument that while eBPF provides the raw visibility, it's the intelligent design of eBPF tools with these three properties that ultimately delivers actionable insights for complex cloud-native environments.

Demo / Proof of Concept

▶ Watch: Key Properties 2 & 3: Structured APIs and Programmability (6:50)

Dom Del Nano provided a compelling demo focusing on Pixie, showcasing how it embodies the three core properties—data enrichment, structured APIs, and programmability—to create a powerful end-to-end observability pipeline. The demonstration highlighted new, upcoming functionality within Pixie, particularly its ability to ingest and correlate diverse data types.

The demo began by illustrating data enrichment. Pixie's service map was displayed, clearly showing Kubernetes service and deployment names rather than raw IP addresses or process IDs. This immediate, high-level context underscores how Pixie automatically enriches low-level eBPF data with relevant Kubernetes metadata, making the system's behavior understandable at an application level.

Next, the role of structured APIs was demonstrated through Pixie's Pixel language. Del Nano pointed out that all the visualizations within Pixie's UI are defined by Pixel code, a Python-like language. This means users aren't locked into predefined dashboards but can manipulate the underlying Pixel script to customize their views and analyses, effectively using the Pixel language as a flexible API for data access and transformation.

The most significant part of the demo focused on programmability, particularly Pixie's new capability to ingest and correlate external data sources. Del Nano presented a scenario involving an application constantly consuming too much memory, leading to frequent OOM (Out Of Memory) kills.

1. Detecting OOM Kills: Initially, a simple BPF trace script was used within Pixie to detect and report OOM kill events. This script checks if the operating system or Kubernetes has issued any kills, providing basic event data.
2. Correlating with Syslog: The groundbreaking feature demonstrated was Pixie's ability to combine these eBPF-derived OOM kill events with syslog output. A new Pixel script was introduced that not only embedded the existing BPF trace code for OOM kills but also leveraged Pixie's new capability to tail log files and convert their content into data frame representations.
3. Merging Data Frames: This advanced Pixel script was designed to merge these two disparate data sources. When an OOM kill event matched, the script would combine the timestamp of the kill, the process ID (PID), the command executed, and the relevant syslog lines from the last five minutes associated with that event. This correlation provides invaluable context for debugging, allowing engineers to see not just that an OOM kill occurred, but also what the system was doing immediately before it.
4. OpenTelemetry (OTel) Export and Visualization: The combined, enriched data was then configured to be sent off to an OpenTelemetry (OTel) export. Del Nano also showcased a new "pipeline flow" visualization within Pixie, which visually represented the data flow: HTTP events, syslog data, and OOM kill tracers being ingested, flowing through Pixie's data collectors, aggregated, and then exported via OTel, in addition to being visualized in the UI. This demonstrated Pixie's architecture as an end-to-end observability pipeline, capable of processing data at the agent level (where eBPF and file tailing occur) and aggregator level, before exporting to external sinks.

This demo effectively illustrated how Pixie, by combining data enrichment, structured APIs, and powerful programmability, can correlate low-level eBPF events with higher-level application and system log data, moving beyond simple auto-instrumentation to provide comprehensive, actionable insights for complex debugging scenarios. It positions Pixie alongside tools like Fluent Bit and Vector in its ability to process and route diverse data sources, but with the unique advantage of kernel-level eBPF visibility.

Defensive Implications

▶ Watch: Project Example: Inspector Gadget for Kubernetes Observability (8:15)

For security practitioners and defenders, the insights from Dom Del Nano's talk underscore the profound impact eBPF has on enhancing security posture in cloud-native environments. The broad, kernel-level visibility offered by eBPF provides an unprecedented vantage point for detecting and responding to threats, going far beyond traditional host-based monitoring.

Firstly, runtime security is significantly bolstered by eBPF. Tools like Pulsar, as highlighted in the talk, demonstrate how eBPF can tap into critical system calls, file system events, and network traffic to enforce granular security policies. Defenders can define rules to detect anomalous behavior, such as unauthorized access to sensitive files like /etc/shadow by processes other than sshd, or the use of insecure protocols and tools like telnet or netcat. This capability allows for real-time threat detection and alerting without requiring agents within application containers or modifications to the application itself.

Secondly, the data enrichment capabilities of eBPF tools are crucial for incident response and forensic analysis. Raw kernel events are often too low-level to be immediately useful in a security context. However, when enriched with Kubernetes metadata (pod names, namespaces, service accounts) and container context, security teams can quickly attribute suspicious activity to specific workloads. This drastically reduces the time to identify compromised assets or vulnerable applications during an incident. The ability to introspect TLS libraries to see plaintext encrypted traffic is particularly powerful, as it allows defenders to uncover malicious communications that would otherwise be hidden.

Thirdly, structured APIs and programmability empower defenders to build custom security detections and integrate eBPF data into their existing security ecosystems. A structured API (like Hubble's gRPC or Pixie's Pixel language) allows security teams to query, filter, and extract specific security-relevant events for integration with SIEMs (Security Information and Event Management), SOAR platforms, or custom threat hunting tools. Programmability, especially as seen in Pulsar's policy engine, means defenders can express complex attack patterns or compliance requirements directly within the eBPF framework, enabling highly targeted and adaptive security monitoring. The ability to correlate eBPF events with other data sources, such as syslog as demonstrated by Pixie, provides a holistic view during investigations, linking kernel-level indicators of compromise with application-level logs for a complete picture of an attack chain.

In essence, eBPF-driven observability tools provide the foundational visibility for robust cloud-native security. Defenders should prioritize tools that effectively implement data enrichment, structured APIs, and programmability to move beyond simple event collection, enabling proactive threat detection, faster incident response, and deeper forensic capabilities in complex, distributed environments.

Key Takeaways

• eBPF for Auto-Instrumentation: eBPF offers broad, language and framework-agnostic auto-instrumentation at the kernel level, significantly reducing the effort and increasing the coverage compared to manual instrumentation in polyglot microservice environments.
• Insights Over Raw Data: While eBPF provides a "firehose" of low-level data, its true value lies in tools that aggregate, analyze, and provide actionable insights, rather than just raw telemetry.
• Three Pillars of Impactful eBPF Tools: Effective eBPF tools must possess data enrichment (adding application/Kubernetes context), structured APIs (for flexible data access and filtering), and programmability (for domain-specific logic and extensibility).
• Higher-Level Tools are Key: Unless you are an eBPF developer, leveraging higher-level eBPF projects (like Pixie, Hubble, Inspector Gadget, Pulsar) is the fastest way to gain valuable insights and avoid the complexity of writing eBPF programs directly.
• Correlation of Diverse Data Sources: Advanced eBPF tools can correlate kernel-level eBPF events (e.g., OOM kills) with other system data (e.g., syslog files) to provide comprehensive context for debugging and security investigations.
• End-to-End Observability Pipelines: eBPF-powered solutions are evolving into full observability pipelines, enabling data processing at the agent and aggregator levels, with flexible export capabilities (e.g., to OpenTelemetry), akin to specialized log processors but with kernel-level visibility.

About the Speaker(s)

Dom Del Nano is the CEO and founder of Cosmic, a company focused on cloud-native observability. He is also a core maintainer for the CNCF Pixie project, an open-source observability platform for Kubernetes. Del Nano's involvement with Pixie began as an end-user approximately three and a half years ago before transitioning into a maintainer role. Prior to his current endeavors, he gained significant experience in the eBPF space while working at CrowdStrike, where he contributed to their eBPF Linux sensor. His background reflects several years of hands-on experience and deep expertise in eBPF technologies and their application in modern monitoring and security.