Observability Pipeline Query Languages: Present and the Future - Jacek Migdal, Quesma

Observability Pipeline Query Languages: Present and the Future

Speakers: Jacek Migdal; Quesma

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=iiI91sUMtdg

Overview

In this insightful talk from KubeCon EU, Jacek Migdal, founder of Quesma, tackles the pervasive fragmentation within observability pipeline query languages. Drawing from his decade-long experience at a major observability vendor and his current work on database gateways, Migdal candidly admits his role in contributing to the very "mess" he now seeks to address. The core problem highlighted is the proliferation of proprietary, vendor-specific query syntaxes for logs, metrics, and traces, which leads to significant challenges such as vendor lock-in, difficulties in multi-vendor environments, and hindered automation efforts.

Migdal argues that this fragmentation prevents organizations from fully leveraging their observability data, stifles innovation, and creates unnecessary complexity for engineers and business users alike. His proposed solution advocates for a standardized approach rooted in SQL, but enhanced with a pipe operator (|) and custom functions to better accommodate the unique, real-time, and ad-hoc nature of observability queries. This evolution aims to provide a unified, expressive, and extensible language that can span all observability signals and even integrate with broader data analytics, ultimately empowering engineers with more efficient debugging tools and enabling advanced automation, including AI-driven insights.

The talk is a call to action for the cloud-native community to embrace a common language, much like OpenTelemetry unified collection, to unlock the full potential of observability data. By demonstrating a practical proof of concept using a Grafana plugin and ClickHouse, Migdal illustrates how a pipe-extended SQL can simplify complex observability tasks, from parsing log lines and enriching data with geo-IP information to integrating threat intelligence, all within a coherent and human-readable syntax. This initiative represents a critical step towards a more interoperable and efficient future for monitoring and troubleshooting complex distributed systems.

Background

▶ Watch: Introduction and the foundational Unix pipe philosophy. (0:00)

The concept of chaining specialized programs together, known as pipes, originated in Unix systems even before Linux. In an era of slow computers, this imperative approach allowed users to combine simple commands like find, grep, sort, unique, and head to perform sophisticated data manipulation. This philosophy of composability proved remarkably resilient and continues to be used for log management on Linux servers today.

The idea of pipes was later commercialized and adapted by Splunk, one of the first full-text commercial vendors for machine data. Splunk recognized the constant need to debug system failures and envisioned a "Google for machine data." They centralized machine data and introduced a query language with a similar pipe-like syntax. Unlike Unix pipes, which are imperative, Splunk's language (and subsequent similar languages like Microsoft Kusto Query Language - KQL, and Elastic Query Language - EQL) are declarative, akin to SQL. Users specify what needs to be done, and the system optimizes how to execute it across distributed systems. This approach became incredibly popular because it was optimized for real-time insights, enabling quick, ad-hoc, and iterative debugging. Its concise nature, familiarity to developers (due to Unix parallels), and integration with visualization tools further cemented its adoption in the log-centric observability world.

Concurrently, another observability paradigm emerged, primarily from Google's internal Borgmon system, which later evolved into Prometheus in the open-source community. Prometheus became the de facto standard for querying metrics. PromQL, Prometheus's query language, is optimized for time-series data, allowing users to select, filter, and aggregate metrics efficiently. Its strength lies in handling naturally time-ordered data points, offering specialized functions for windowing, interpolation, and rate calculations, which are crucial for assessing service health in YAML configurations.

However, the observability landscape exploded with the rise of venture capital funding. Companies like Splunk, which raised $50 million and reached a market cap of $47 billion, demonstrated the immense value in this sector. This success, coupled with the advent of OpenTelemetry and the Cloud Native Computing Foundation (CNCF), lowered the barrier to entry for new observability startups. Instead of building everything from scratch, companies could leverage existing open-source collection and storage engines. While this fostered innovation, it also led to a significant problem: a proliferation of proprietary query languages.

Unlike the analytical database world, where SQL remains the dominant standard with minor dialects, the observability space became a "mess" of incompatible syntaxes. Migdal illustrates this with a simple example: parsing an application name from logs and performing an aggregation yields different syntax across almost every vendor. Worse, some vendors operate on tokens, leading to inconsistent search results (e.g., searching "invalid type" might not find "invalid type error" without a wildcard). This fragmentation results in:

• Vendor lock-in: Customers are tied to a specific vendor's language.
• Limited advanced feature adoption: Users avoid complex features due to the learning curve and fear of lock-in.
• Difficulty in multi-vendor environments: Large organizations often use several observability tools, requiring engineers to switch contexts and learn multiple languages.
• Challenges for automation and AI: AI agents or threat intelligence systems struggle to integrate and automate across diverse, incompatible query languages.
• Context switching: Engineers constantly jump between logs, metrics, traces, and even external databases (e.g., to join user IDs with customer data) due to lack of a unified query mechanism.

Migdal points to the data world's journey, where NoSQL databases gained traction but eventually, many of their innovations (like JSON parsing) were adopted by SQL. SQL, a language created 50 years ago, has shown remarkable adaptability and resilience, even adding graph support. This historical pattern suggests that rather than inventing entirely new languages, evolving a proven standard like SQL might be the most sustainable path forward for observability.

Key Findings

▶ Watch: Key reasons for log-centric language popularity. (3:30)

The central findings of Jacek Migdal's talk revolve around diagnosing the current state of observability query languages and proposing a pragmatic path forward:

1. Severe Fragmentation: The observability market is characterized by a "mess" of proprietary, vendor-specific query languages for logs, metrics, and traces. This fragmentation leads to vendor lock-in, limits the adoption of advanced features, complicates multi-vendor deployments, and creates significant hurdles for automation and AI integration.
2. SQL's Enduring Relevance: Despite the rise of domain-specific languages, SQL remains a robust, evolving, 50-year-old standard in the data world. Its ability to adapt to new paradigms (like JSON parsing and graph support) makes it a strong candidate for unifying observability queries, leveraging its vast existing user base and tooling ecosystem.
3. SQL's Observability Gaps: Traditional SQL, while powerful, is often verbose, non-intuitive, and challenging for common, real-time observability tasks. Operations like calculating rate from cumulative metrics or finding top N with others in log analytics require complex window functions and multiple subqueries, making the queries long, unreadable, and prone to edge-case errors.
4. CNCF's Semantic Alignment: The Cloud Native Computing Foundation (CNCF) has been actively working on unifying observability languages. After extensive research and interviews with DSL designers, the conclusion is to adopt SQL semantics as a foundational step. This ensures consistent meaning of operations and data types, even if the syntax continues to evolve.
5. The Pipe Operator as a Solution: To address SQL's syntactical shortcomings for observability, Migdal advocates for extending SQL with a pipe operator (|). This operator allows for chaining operations in a more logical, left-to-right flow, mirroring the familiarity of Unix pipes and existing observability query languages like Splunk's SPL. This "syntactical sugar" greatly enhances readability and composability.
6. Extensibility and Custom Operators: The pipe operator provides a natural and logical place to introduce custom operators and functions tailored for observability (e.g., rate(), parse_log(), enrich_geoip(), enrich_threat_intel()). These custom operators can either be transpiled into complex standard SQL for immediate compatibility with existing engines or be natively supported by specialized observability databases for performance gains.
7. Proof of Concept Validation: Quesma has developed an open-source Grafana plugin leveraging ClickHouse to demonstrate the viability of pipe-extended SQL. This prototype showcases how complex parsing, data enrichment, and filtering tasks can be performed intuitively and efficiently within this proposed language, validating its practical application.

Technical Deep Dive

▶ Watch: Introduction of PromQL for metric-centric time series. (4:40)

The core of Jacek Migdal's technical argument rests on the paradox of SQL: it's a powerful, universally understood language, yet it struggles with the specific demands of observability. He identifies several key areas where traditional SQL falls short and how his proposed pipe-extended SQL addresses these limitations.

Challenges of Traditional SQL for Observability:

1. Performance and Real-time Analytics: While modern SQL engines are highly optimized, the rapid, ad-hoc nature of observability debugging often demands immediate answers. Traditional SQL, especially when queries become complex, can be perceived as too slow or cumbersome for critical outage scenarios where "quick answers" are paramount, even if approximate. The speaker notes that in observability, slight inaccuracies (e.g., "million errors or million two errors") are often acceptable if they lead to faster insights, unlike financial reporting where precision is absolute.
2. Complexity for Common Observability Tasks:

• Rate Calculation: A fundamental operation in metric analysis, converting cumulative metrics into a rate (e.g., requests per second). In PromQL, this is a concise rate() function. In standard SQL, as Migdal illustrates, it requires advanced window functions (LAG()), careful handling of edge cases like process restarts (metric overturns), and missing data points. The resulting SQL query is "long," "not readable," and "does not express your intent," often requiring multiple subqueries and still potentially missing edge conditions for the first/last metric. Even generative AI struggles to produce correct SQL for this without multiple refinements.
• Top N with Others: In log analytics, finding the top 10 most frequent errors (e.g., by container app) and grouping all other less frequent errors into an "others" category is a common requirement. In SQL, this typically necessitates three subqueries, again breaking the "beauty of SQL" by focusing on implementation mechanics rather than intent.
• Data Enrichment and Macros: Observability often benefits from on-the-fly enrichment, such as mapping an IP address to a country or checking against a threat intelligence feed. Integrating these "macros" or external data lookups cleanly into standard SQL is challenging. While some vendors allow uploading reference files, this can lead to eventual consistency issues.

The Solution: SQL with Pipe Operator and Custom Extensions

Migdal's proposal, already being discussed and partially implemented in major platforms like Google BigQuery and Databricks Firebolt, is to augment SQL with a pipe operator (|). This is not merely syntactic sugar; it fundamentally changes how observability queries can be structured and extended.

1. Logical Query Flow: The pipe operator allows chaining operations in a linear, left-to-right fashion, similar to how data flows through Unix pipes. This addresses SQL's often non-intuitive ordering of clauses (WHERE, HAVING, QUALIFY) and the nesting required for subqueries or Common Table Expressions (CTEs).

• Example: Instead of SELECT ... FROM (SELECT ... FROM ... WHERE ...) AS subquery, one could write FROM ... WHERE ... | SELECT .... This makes the query flow more intuitive and readable, expressing the transformation of data step-by-step.

1. Extensibility through Custom Operators: The pipe operator provides a "more logical place" to introduce custom functions and operators specific to observability. These can be:

• Domain-specific operators: Like rate() for metrics, or top_n_others() for logs, which abstract away the underlying complex SQL window functions or subqueries.
• Parsing operators: Such as EXTRACT_HOST() or EXTRACT_IP() to parse unstructured log text into structured fields.
• Enrichment operators: Like ENRICH_GEOIP() or ENRICH_THREAT_INTEL() to join with external data sources or APIs at query time.

1. Transpilation and Native Support:

• Transpilation: A key advantage is that these custom pipe operators can initially be transpiled into complex standard SQL queries that existing SQL engines (like ClickHouse, Postgres, etc.) can execute. This provides immediate compatibility and allows for rapid adoption without requiring fundamental changes to database engines. The speaker notes that "writing like a transpiling is really fast."
• Native Support: Over time, as the standard evolves, database engines could add native support for these operators, potentially leading to more efficient execution and specialized optimizations for observability workloads.

1. Semantic Foundation: The proposal strongly emphasizes adopting SQL semantics first. This means ensuring that underlying operations, data types, and null handling behave consistently with SQL standards. While the syntax can be flexible and evolve through community experimentation, a shared semantic understanding is crucial for true unification.

This approach combines the robust, standardized foundation of SQL with the intuitive, extensible nature required for real-time observability. It allows for the creation of a powerful, unified language that can be understood by both humans and AI agents, fostering a more interoperable and automated future for cloud-native systems.

Demo / Proof of Concept

▶ Watch: The emerging challenge of many diverse query languages. (6:09)

Jacek Migdal presented a tangible proof of concept to illustrate the practical application of pipe-extended SQL for observability. This demo, developed as an open-source Grafana plugin by Quesma, showcases how the proposed language can simplify complex tasks using ClickHouse as the backend data store.

The demonstration scenario involved analyzing OpenSSH logs to detect potential brute-force attacks. The key steps and features highlighted were:

1. Initial Query and Implicit Context: The demo started with a basic query on OpenSSH logs. A significant aspect is that the time range for the query is implicitly set by the Grafana dashboard, a common practice in observability UIs, eliminating the need for explicit time filters in the query itself.

1. Parsing Unstructured Logs:

• The initial log lines contained valuable information, but it was embedded within unstructured text.
• Instead of writing complex subqueries or pre-processing the data, the user could extend the query using the pipe operator: | EXTRACT_HOST(text) AS extracted_host, EXTRACT_IP(text) AS extracted_ip.
• This custom EXTRACT_HOST and EXTRACT_IP operator parsed the relevant hostnames and IP addresses directly from the log text, adding them as new columns (extracted_host, extracted_ip) to the result set. This elegantly solves a common problem in log analytics without verbose SQL.

1. Data Enrichment - IP Geolocation:

• To gain more context about the source of the SSH attempts, the demo then applied another custom operator for IP geolocation: | ENRICH_GEOIP(extracted_ip) AS country.
• Behind the scenes, this operator would perform a lookup (e.g., download an IP-to-country database) and join the results, adding a country column to the data. This is a powerful feature for identifying the geographical origin of suspicious activity, which would be challenging and inconsistent to implement directly in traditional SQL due to the need for external data.

1. Data Enrichment - Threat Intelligence:

• Further refining the investigation, the demo showed how to integrate threat intelligence: | ENRICH_THREAT_INTEL(extracted_ip) AS is_bot.
• This operator queried an external threat intelligence vendor's API to determine if the extracted IP addresses were associated with known bots or malicious actors, adding an is_bot boolean column.
• The ability to chain such enrichment operations directly within the query language is highly beneficial for security analysts, allowing them to quickly pivot and filter based on external context.

1. Filtering and Analysis:

• With the enriched data, the user could then apply standard SQL-like filtering, such as WHERE is_bot = TRUE AND country = 'China', to narrow down the results to specific threats or regions.
• The demo concluded by showing how this unified language allows for a seamless workflow, from raw log lines to highly contextualized and actionable insights, all expressed in a concise and readable manner.

The open-source nature of the Grafana plugin allows the community to "play with it," "modify and play with the syntax," and contribute to the evolution of this language. Migdal emphasizes that "languages and syntaxes are not written by committee, they're like written by people like you." This practical implementation serves as a strong argument for the feasibility and benefits of pipe-extended SQL, demonstrating that complex, real-world observability challenges can be addressed with a more intuitive and extensible query language.

Defensive Implications

▶ Watch: Explosive growth and venture success in observability. (6:15)

The adoption of a standardized, pipe-extended SQL for observability has profound implications for cybersecurity defenders, offering significant improvements in efficiency, automation, and threat detection capabilities.

1. Standardized Threat Hunting and Incident Response: With a unified query language, security analysts can develop standardized threat hunting playbooks and incident response procedures that are portable across different observability platforms. This means a query written to detect a specific attack pattern (e.g., parse_logs() | ENRICH_THREAT_INTEL() | WHERE is_malicious = TRUE) can be executed consistently, regardless of whether the underlying data resides in Splunk, Elastic, Prometheus, or a custom log store. This reduces the learning curve for new tools and accelerates response times during critical incidents.

1. Enhanced Security Automation and SOAR Integration: The fragmented nature of current observability query languages makes it difficult to build robust automation. A standardized language, acting as a "protocol," enables Security Orchestration, Automation, and Response (SOAR) platforms, custom scripts, and AI agents to seamlessly query and manipulate observability data from various sources. This can automate tasks like:

• Automatically enriching alerts with geo-IP or threat intelligence data.
• Triggering defensive actions based on specific log patterns or metric anomalies.
• Feeding observability data into security information and event management (SIEM) systems with consistent context.

1. Vendor-Agnostic Security Tooling: A common language fosters the development of vendor-agnostic security tools. Instead of building integrations for each proprietary query language, security vendors and open-source projects can target a single, standardized interface. This encourages innovation in security analytics, threat detection, and compliance reporting that is not tied to a specific observability provider, giving organizations more flexibility and choice.

1. Improved Contextual Analysis: The ability to chain enrichment operators directly within queries (e.g., ENRICH_GEOIP(), ENRICH_THREAT_INTEL()) allows defenders to immediately add crucial context to security events. This means an IP address in a log can be instantly linked to its country of origin, known malicious reputation, or even internal user/asset information, without requiring manual lookups or complex, out-of-band joins. This holistic view significantly speeds up investigations and provides richer data for decision-making.

1. Reduced Learning Curve for Security Analysts: Many security professionals are already familiar with SQL from database interactions. Adopting a SQL-based language with intuitive extensions lowers the barrier to entry for querying complex observability data. This allows security teams to become more self-sufficient in exploring logs, metrics, and traces for security insights, reducing reliance on specialized observability experts and freeing up resources.

1. Better AI Training and Supervision: As AI increasingly assists in security operations, a formal, human-readable query language is vital. It provides a clear target for AI agents to generate queries and allows human analysts to easily supervise and validate the AI's output. Furthermore, the pipe-based structure enables AI to provide feedback on partial query results, guiding analysts (or other AIs) to refine their investigations more effectively. The ability to collect "here were the queries that we found the issue" becomes invaluable for training and evaluating AI models.

In essence, a unified observability query language transforms observability data from a disparate collection of signals into a coherent, actionable intelligence source for defenders, making security operations more automated, efficient, and effective against evolving threats.

Key Takeaways

• Observability Query Language Fragmentation is a Major Problem: The proliferation of proprietary query languages across different observability vendors leads to vendor lock-in, hinders innovation, complicates multi-vendor environments, and creates significant barriers for automation and AI.
• SQL is the Foundation for Unification: Despite its age, SQL is a highly adaptable and widely understood standard. The Cloud Native Computing Foundation (CNCF) recommends adopting SQL semantics as the basis for a unified observability query language.
• Traditional SQL Needs Observability-Specific Enhancements: While powerful, standard SQL is often too verbose and non-intuitive for common, real-time observability tasks like calculating rates from metrics or performing top-N aggregations with an "others" category.
• The Pipe Operator Enhances SQL for Observability: Introducing a pipe operator (|) allows for a more logical, left-to-right chaining of operations, improving readability and providing a natural extension point for custom, domain-specific functions (e.g., rate(), parse_log(), enrich_geoip(), enrich_threat_intel()).
• Practical Implementation is Feasible and Beneficial: Quesma's open-source Grafana plugin, leveraging ClickHouse, demonstrates how pipe-extended SQL can effectively parse, enrich, and filter observability data, abstracting complex SQL into intuitive custom operators.
• A Unified Language Benefits Everyone: This standardization will empower engineers with better debugging tools, facilitate robust automation and AI-driven insights, reduce vendor lock-in, and provide a common "protocol" for building future observability and security tools.

About the Speaker(s)

Jacek Migdal is the founder of Quesma, a startup focused on database gateways. He brings a wealth of experience in the observability space, having worked for ten years at a prominent observability vendor, a period he humorously acknowledges as contributing to the "mess" of fragmented query languages he now aims to solve. Prior to his entrepreneurial venture, Migdal also held positions at leading technology companies including Nvidia, Qualcomm, and Meta. His extensive background in distributed systems, databases, and observability uniquely positions him to advocate for a standardized, SQL-based approach to querying observability data. He is a strong proponent of SQL, viewing its 50-year survival and adaptability as a testament to its enduring value.