Beyond CloudEvents: Endpoints, Messages, Schemas – CNCF XRegistry - Manuel Ottlik, HDI Global SE

Beyond CloudEvents: Endpoints, Messages, Schemas – CNCF XRegistry

Speakers: Manuel Ottlik, Product Owner for Cloud Native Platforms, HDI Global SE

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=M56SHzETAmM

Overview

This talk, presented by Manuel Ottlik of HDI Global SE, delves into the evolution of event-driven architecture tooling within the CNCF ecosystem, specifically focusing on CloudEvents and the newer XRegistry project. While CloudEvents successfully standardized the metadata for events, enabling interoperability across diverse messaging systems, it left a critical gap: the discovery and management of what events exist, what their payloads look like, and where they can be sent or received. XRegistry emerged to fill this void, providing a vendor-agnostic, extensible framework for managing metadata about any resource, with a particular focus on event-driven architecture entities like messages, schemas, and endpoints.

Manuel Ottlik, a maintainer of both projects, aims to bridge the knowledge gap for both newcomers and veterans. He highlights XRegistry's core value proposition: bringing order and discoverability to increasingly complex distributed systems by centralizing the definition and versioning of critical integration assets. For end-user companies like HDI Global SE, XRegistry promises to harmonize schemas across disparate integration products, reduce duplication, and foster a more connected and understandable metadata landscape.

The talk underscores the importance of standardized approaches to managing the entire lifecycle of event-driven interactions, moving beyond just the event "envelope" to encompass the full context of messages, their content, and their communication channels. XRegistry represents a significant step towards achieving comprehensive interoperability and governance in modern cloud-native environments, offering a foundational layer for platform engineering teams to build robust internal developer platforms.

Background

▶ Watch: Introduction and talk agenda overview (0:00)

The journey towards XRegistry begins with CloudEvents, a CNCF graduated project initiated in 2018. CloudEvents addresses the fundamental challenge of interoperability in event-driven systems by defining common metadata for events. Manuel Ottlik uses the analogy of sending a letter: CloudEvents doesn't dictate how to write the letter (the event payload), but rather what information should be on the envelope (the event metadata). This ensures that regardless of the underlying messaging protocol or event broker, consumers can consistently interpret essential details like the event's source, type, and unique ID. The project boasts nine SDKs and six protocol bindings, including AMQP, Kafka, HTTP, and MQTT, supporting both binary and structured modes for embedding metadata. The binary mode allows for non-breaking changes to existing consumers, while the structured mode integrates metadata directly into the payload.

CloudEvents continues to evolve with powerful extensions. CESQL (CloudEvents Structured Query Language), now at version 1.0.0, enables sophisticated filtering of events based on their metadata, a feature already adopted by KNative. Four new extensions released in the last year further enhance its capabilities:

• BAM (Business Activity Monitoring): Defines headers for monitoring the progress of business processes.
• Data Classification: Addresses concerns like data confidentiality, regulation (e.g., GDPR), and category (e.g., sensitive/non-sensitive).
• Deprecation: Provides critical information about an event type's deprecation status, support timeline, and migration path.
• OPC-UA: Offers a mapping for the industrial protocol's headers to CloudEvents, adding domain-specific attributes.

Despite CloudEvents' success in standardizing event metadata, it became apparent that a broader set of questions remained unanswered in complex event-driven ecosystems. Returning to the letter analogy, users still needed to know:

1. What event types (letters/envelopes) actually exist? And what metadata and extensions do they use?
2. What does the actual letter inside look like? What is the structure or schema of the event payload?
3. Where are the post boxes? In distributed systems, where are the endpoints to send and receive these messages?

These questions initially arose within the CNCF Serverless Working Group, the same group that created CloudEvents. Their initial attempt to address these concerns was called CloudEvents Discovery, which defined three core entities:

• Message Definitions: Describing event types and their versions.
• Schemas: Defining the actual payload structure (the "letter").
• Endpoints: Specifying where messages can be sent or received.

However, recognizing that these three entities shared a common pattern—managing versioned metadata about resources—the group decided against creating three separate specifications. Instead, they aimed for a single, unified core specification that could be extended to cover these and potentially other types of resources. This led to the creation of XRegistry, standing for Extensible Registry. XRegistry's core specification is deliberately abstract, designed to manage metadata about any resource, making it a powerful foundation for diverse registry needs beyond just eventing.

Key Findings

▶ Watch: CloudEvents structure: binary and structured modes (2:00)

The core innovation of XRegistry lies in its abstract, extensible core specification. Unlike CloudEvents, which is specific to event metadata, XRegistry provides a generic model for managing metadata about any resource. This model is elegantly simple: groups contain resources, and those resources can be versioned. This fundamental structure allows XRegistry to be highly adaptable.

A primary design goal for XRegistry is to be vendor agnostic. The specification is crafted such that it can be adopted by any vendor without requiring them to fundamentally alter their business model. This commitment to openness fosters broader adoption and avoids vendor lock-in, which is crucial for interoperability in the cloud-native landscape.

The main value proposition of XRegistry is the interoperability it provides through its core specification. By offering a common language and structure for defining and managing resource metadata, it enables different systems, tools, and organizations to understand and interact with each other's registries seamlessly. This is a significant step towards a more harmonized and discoverable ecosystem.

The Endpoints, Messages, and Schemas discussed earlier are not separate projects but rather specific implementations of the XRegistry core specification. They demonstrate how the abstract groups -> resources -> versions model can be concretely applied to the needs of event-driven architectures. For instance, schema groups are groups, schemas are resources, and schema versions are versions within the XRegistry model.

To accelerate adoption and provide practical examples, the XRegistry project includes a server implementation (a reference implementation written by Duck Davis) and a CLI (developed by Clemens Vastas). These tools allow users to interact with and manage XRegistry instances, whether for testing, development, or production use.

A key architectural principle of XRegistry is the symmetry between its representations. Users can define their registry data in a simple JSON file, push it to a REST API-based server, and then export it back into a file format. Furthermore, XRegistry supports serving registry data via a static file server, enabling read-only access for scenarios where a full API server might be overkill, such as platform engineering teams providing schema definitions via a GitHub repository and a pipeline.

Manuel Ottlik illustrates the profound impact of XRegistry through the lens of an end-user company like HDI Global SE. With 300 developers managing diverse interfaces across API management, message buses, and event brokers, schema repetition and inconsistency are rampant. XRegistry offers a solution by providing a single, centralized place to store and manage all schema definitions (e.g., JSON Schema, Avro, Protobuf, OpenAPI, AsyncAPI). This centralization allows for:

• Avoiding schema repetition: Schemas are defined once and referenced across different integration products.
• Harmonizing schemas: Ensuring consistency in business object and event definitions across the enterprise.
• Connecting metadata: Linking previously siloed information about business objects and events, even extending to data products in BI platforms.

By abstracting schema definitions away from specific integration products, XRegistry enables a more agile and governed approach to managing enterprise-wide data and event contracts.

Technical Deep Dive

▶ Watch: Latest CloudEvents extensions: CESQL, BAM, Deprecation (3:10)

The XRegistry core specification is built upon a simple yet powerful meta-model: groups containing resources, which in turn can have versions. This hierarchical structure provides a flexible framework for organizing and managing diverse metadata.

For Endpoints, an XRegistry entry might define a group (e.g., com.example.nuts) containing an endpoint resource (e.g., producer.example.com). Each endpoint version would then specify details like:

• name: A unique identifier for the endpoint.
• type: Whether it's a producer or consumer.
• envelope: The event envelope specification used, such as cloud events 1.0.
• protocol: The messaging protocol, e.g., nats or http.
• protocol options: Protocol-specific configuration, like a URI for NATS or details about the subject structure.

Message Definitions are structured within message groups, which contain individual message resources, each with its own version. A message version typically includes:

• envelope: The CloudEvents version (e.g., 1.0).
• envelope metadata: Specific, concrete values for CloudEvents attributes, such as spec version (1.0), type (e.g., com.example.task.created), and source (e.g., com.example.todo-list-service). This allows consumers to precisely filter and interpret events.
• data schema format: The format of the event payload schema (e.g., JSON Schema).
• data schema URI: A reference to the specific schema version defined in the XRegistry's schema section.

Schemas are organized into schema groups, containing schema resources, each with its own version. A schema version holds the actual schema definition, which can be any supported format:

• The actual JSON Schema object.
• Avro, Protobuf, OpenAPI, or AsyncAPI definitions.

The XRegistry server implementation extends these basic definitions with additional metadata when exposed via its REST API. For any resource, attributes like self URL, a unique X ID, epoch timestamps (created at, modified at), and navigational URLs are automatically added. This enriches the resource representation with operational context and discoverability. The server also handles default versioning, meaning that when querying a resource, the most up-to-date version is automatically integrated into the resource's default representation, though all historical versions remain accessible.

A crucial aspect of XRegistry's design is its ability to support custom models. The talk demonstrates this by showing how the XRegistry server can host different registry models beyond just messages, schemas, and endpoints. Examples include:

• APIs.guru model: Groups of API providers containing APIs with their own attributes.
• Doc Store model: Groups of documents containing formats with specific attributes.

This flexibility proves that XRegistry is truly an abstract framework, capable of managing metadata for virtually any domain.

The project emphasizes symmetrical representations across different interaction modes:

1. JSON File: Registries can be defined in simple JSON files, making them version-controllable and easy to share.
2. REST API: The XRegistry server provides a full RESTful interface for programmatic interaction, including curl commands for adding and retrieving resources.
3. Static File Server: For read-only access, a registry can be split into individual files mimicking the REST API structure and hosted on any static file server. This is useful for scenarios where a platform team publishes schemas that developers consume.

Finally, XRegistry seamlessly integrates with CloudEvents. The data schema field within a CloudEvent can directly reference a schema definition stored within an XRegistry instance, closing the loop between event metadata and payload structure.

Demo / Proof of Concept

▶ Watch: XRegistry's core entities: messages, schemas, endpoints (6:00)

Manuel Ottlik provided a live demonstration using a to-do list service example to illustrate XRegistry in action. The scenario involved a simple application that creates "task created" and "task completed" events.

The architecture for the demo comprised:

• A to-do list service (the producer).
• An event broker, specifically NATS, to which events are emitted.
• A consumer service that processes these events.
• An XRegistry instance (the reference server implementation) sitting above, providing the definitions for the events, schemas, and endpoints.

The demonstration began by showcasing a simple JSON file representing an XRegistry. This file contained the definitions for:

• Endpoints: An entry for com.example.nuts as a producer, specifying cloud events 1.0 as the envelope, nats as the protocol, and protocol options including a URI and subject structure for NATS.
• Message Groups: A group com.example.todo-list containing two messages: task.created and task.completed. Each message version specified envelope metadata like spec version (1.0), type (e.g., com.example.task.created), and source (com.example.todo-list-service), along with a data schema URI pointing to the relevant schema.
• Schema Groups: A group com.example.todo-list containing a task schema. The schema version for this task defined a JSON Schema object with properties like id, title, and completed.

Manuel then used a curl command to post this JSON file to the REST API of the XRegistry server implementation. Once loaded, he navigated to the server's UI, which Duck Davis created. The UI displayed the registered endpoints, message groups, and schema groups. He highlighted how the UI shows additional attributes like self URL, X ID, and epoch timestamps (created at, modified at) that are automatically added by the server, enriching the metadata.

He demonstrated how to view different versions of a message or schema, explaining that the default view for a resource automatically integrates the most up-to-date version, but all historical versions are accessible. The flexibility of XRegistry's model was further illustrated by showing pre-loaded custom models in the UI, such as the APIs.guru model (for API providers and APIs) and a Doc Store model (for documents and formats), proving that the core spec can be applied to diverse resource types.

Finally, the demo showcased the symmetrical representations by using the UI's export functionalities (inline, doc view, export) to retrieve the registry data back into a file-like structure, demonstrating how server-added metadata can be selectively included or removed for different export purposes. This reinforces the idea that users can seamlessly move between file-based definitions, REST API interactions, and static file server deployments.

Defensive Implications

▶ Watch: XRegistry: Abstract, extensible metadata management for resources (7:00)

While XRegistry is not a security-focused project in its primary design, its contributions to clarity, governance, and standardization within event-driven architectures have significant indirect defensive implications for organizations.

Firstly, by centralizing and harmonizing schema definitions across an enterprise's integration landscape (API management, message bus, event broker, data platforms), XRegistry drastically reduces ambiguity. This clarity is invaluable for threat modeling, as security teams can precisely understand the structure and expected content of events and messages flowing through their systems. Knowing the exact schema for a task.created event, for instance, allows for more accurate identification of malformed or malicious payloads that deviate from the expected contract.

Secondly, the explicit management of event types and their versions helps in maintaining a secure posture over time. The CloudEvents Deprecation extension, which XRegistry can house definitions for, directly aids in managing the lifecycle of event contracts. This ensures that systems are not unknowingly relying on outdated or vulnerable schema versions. A well-governed deprecation process prevents security gaps that might arise from legacy systems interacting with new ones using incompatible or unpatched data structures.

Thirdly, the CloudEvents Data Classification extension is directly relevant to security and compliance, particularly for regulations like GDPR. XRegistry provides a central place to define and associate data classification attributes (confidentiality, regulation, category) with specific event types and their underlying schemas. This enables organizations to:

• Enforce data handling policies: By knowing which events carry sensitive data, appropriate encryption, access controls, and logging can be applied.
• Improve auditing: During an incident or compliance audit, it's easier to trace how classified data was handled across the event-driven ecosystem.
• Reduce data leakage risks: A clear understanding of data sensitivity helps prevent accidental exposure.

Finally, the explicit registration of endpoints (where events are sent/received) provides a comprehensive inventory of communication channels. This is critical for network segmentation, firewall rule management, and monitoring. Security teams can gain a clear picture of event flow, identify unauthorized endpoints, and ensure that only legitimate producers and consumers are interacting with the event fabric. This improved visibility aids in anomaly detection and incident response by providing a definitive map of the event landscape.

In essence, XRegistry contributes to a stronger defensive posture by making the complex world of event-driven microservices more transparent and manageable, thereby reducing the attack surface stemming from ambiguity, inconsistency, and lack of governance.

Key Takeaways

• CloudEvents Continues to Evolve: CloudEvents provides common event metadata, with recent significant additions including CESQL (1.0.0 for filtering), BAM (Business Activity Monitoring), Data Classification (for confidentiality/regulation), Deprecation (for managing event lifecycle), and OPC-UA (for industrial protocol mapping).
• XRegistry Addresses Core Gaps: Beyond CloudEvents' envelope metadata, XRegistry solves the critical problems of discovering what event types exist, what their payloads (schemas) look like, and where they can be sent or received (endpoints).
• Extensible Core Specification: XRegistry's foundation is a vendor-agnostic, abstract model (groups -> resources -> versions) designed to manage metadata about any resource, not just events. This makes it highly adaptable for diverse registry needs.
• Unified Event-Driven Architecture Management: Endpoints, Messages, and Schemas are concrete implementations of the XRegistry core spec, enabling a single, coherent view and management plane for all aspects of an event-driven system.
• Harmonization and Repetition Avoidance: For end-user companies like HDI Global SE, XRegistry provides a central repository to define and harmonize schemas across different integration products (API management, message buses, event brokers), significantly reducing schema duplication and inconsistency.
• Project Maturity & Roadmap: XRegistry has a server reference implementation, a CLI, and supports symmetrical file, REST API, and static file server representations. Release Candidate 1 is available for review, with a 1.0 release targeted for Q3 2024.

About the Speaker(s)

Manuel Ottlik is a Product Owner for Cloud Native Platforms at HDI Global SE, an insurance company based in northern Germany. He is actively involved in the cloud-native community as a maintainer for both the CloudEvents and XRegistry projects, demonstrating his deep expertise and commitment to advancing interoperability and governance in event-driven architectures. His work at HDI Global SE provides valuable real-world context and drives the development of practical solutions for complex enterprise environments.