Production, Consumption, and the Data: The Open Source Security Sandwich

Production, Consumption, and the Data: The Open Source Security Sandwich

Speakers: Mike Lieberman, Co-founder and CTO, Casari

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=vOTl_gjaeLI

Overview

In his insightful VulnCon presentation, "Production, Consumption, and the Data: The Open Source Security Sandwich," Mike Lieberman, Co-founder and CTO of Casari, dissected the pervasive and increasingly complex challenges of securing the modern software supply chain. Lieberman, a prominent figure in the open-source security community and a maintainer of critical projects like GUAC and Salsa, framed the current state of software security as a "dumpster fire," characterized by a relentless stream of vulnerabilities and sophisticated attacks such as the XZ Utils backdoor and SolarWinds. The talk aimed to demystify software supply chain security, defining it as the comprehensive act of securing both the production and consumption of software throughout the entire Software Development Lifecycle (SDLC).

Lieberman introduced the "Open Source Security Sandwich" as a metaphorical framework to conceptualize a holistic approach to supply chain defense. This model emphasizes a layered strategy, from securing the initial creation of software to its eventual consumption, underpinned by robust data collection and analysis. The core message revolved around the critical need for explicit metadata generation and consumption, advocating for open standards and tools to bring order to the chaos of interconnected dependencies. The presentation highlighted the transformative potential of projects like GUAC (Graph for Understanding Artifact Composition) in aggregating diverse security metadata to provide actionable insights for defenders.

The talk is particularly relevant in today's threat landscape, where organizations grapple with an ever-expanding attack surface stemming from their reliance on open-source components and complex build pipelines. Lieberman's expertise, drawn from his extensive involvement with the OpenSSF Technical Advisory Council, CNCF TAG Security, and his co-authorship of "Securing the Software Supply Chain," lends significant weight to his recommendations. By illustrating the myriad attack vectors and presenting a structured, data-centric solution, the talk empowers security professionals to move beyond reactive vulnerability management towards a proactive, observable, and resilient software supply chain posture.

Background

▶ Watch: Introduction and speaker's open source involvement (0:00)

The journey into software supply chain security begins with understanding its fundamental definition: the act of securing the production and consumption of software – in essence, safeguarding the entire SDLC. Every organization producing software contributes to someone else's supply chain, and every organization consuming software pulls external components into its own. This interconnectedness creates a vast and intricate web of dependencies, making the task of ensuring security a monumental challenge. The problem is compounded by the "turtles all the way down" phenomenon, where dependencies have their own dependencies, extending not just into software layers but also hardware and external, often uncontrolled, systems like public package repositories such as PyPI and Maven Central.

Lieberman meticulously outlined the various points of compromise within the SDLC, illustrating each with historical examples:

• Producer: The developer themselves can be malicious, as seen in the XZ Utils incident where a likely state-sponsored actor injected malicious code into the repository, modifying build processes to introduce a backdoor into OpenSSH. Even unintentional mistakes can lead to vulnerabilities, such as the OpenSSH regression attack from 15-20 years ago, which inadvertently reduced the key space required for brute-force attacks.
• Source Code Management (SCM): Repositories can be compromised. The Codecov attack serves as an example where malicious actors gained access to customer credentials and secret keys.
• External Build Parameters: Insecure inputs to the build process can be exploited. The Debian OpenSSL bug, also from 15-20 years ago, resulted from improper entropy generation, severely weakening cryptographic keys.
• Build Process: The build system itself can be hijacked. The SolarWinds attack demonstrated how a sophisticated actor could compromise the build system to inject malicious DLLs into legitimate, signed binaries, making detection extremely difficult.
• Artifact Publication: The process of publishing artifacts can be subverted. Frequent incidents involve malicious npm packages designed for cryptocurrency mining or stealing digital wallet credentials, often published after attackers steal maintainer credentials.
• Distribution Channel: The repositories or channels used to distribute software can be compromised. An old RubyGems vulnerability allowed unauthorized publication of malicious software due to tooling flaws.
• Package Selection: Users can be tricked into downloading malicious lookalikes. The typosquatting issue with request versus requests in the Python ecosystem saw users inadvertently downloading malicious code by mistyping a popular library name.

The sheer scale of this problem is daunting. Lieberman cited an example of analyzing Kubernetes' full dependency tree, including operating system and Go compiler dependencies, extending "20 or 30 layers deep" and involving thousands of packages. Manually tracking and verifying every component and its provenance is practically impossible for most organizations. However, the talk also acknowledged significant progress, with growing community, industry, and government efforts (like the Cyber Resilience Act - CRA) dedicated to bringing order to this chaos through white papers, books, and a proliferation of new security tools.

Key Findings

▶ Watch: Defining software supply chain security and Salsa framework (3:15)

Lieberman's talk underscored several critical findings that collectively paint a path forward for securing the software supply chain:

1. Security is About Observability and Data: At its core, effective software supply chain security hinges on the ability to track and understand what is happening within systems. This means collecting sufficient metadata about software as it moves from development to consumption, enabling organizations to determine "who did what and when." Without this explicit data, identifying vulnerabilities or compromises and planning remediation becomes exceedingly difficult, if not impossible.

1. Open Specifications are Paramount: The adoption of open, standardized specifications for security metadata is crucial. Proprietary formats hinder interoperability and broader community support. Lieberman advocated for:

• SBOMs (Software Bill of Materials): A standardized way to communicate software composition. While acknowledging their current limitations (misunderstanding, immature tooling, over-broad application), he stressed their importance as a "rallying call" and a machine-readable foundation.
• Salsa (Software Supply Chain Levels for Artifacts): Best practices and a framework for providing provenance around software, ensuring that source code matches the built artifact.
• VEX (Vulnerability Exploitability eXchange): A standardized format for communicating the exploitability status of known vulnerabilities in specific software versions.
• OpenTelemetry, Scorecard, GUAC, OCAL, S2C2F, In-Toto Layouts, TUF: A suite of open standards and tools that collectively provide comprehensive visibility and control.

1. The "Open Source Security Sandwich" Provides a Holistic Framework: This metaphor structures the problem and solution into three distinct layers:

• Production (Top Bun): Focused on securing the creation of software using tools like Salsa, VEX, Scorecard, Allstar, and Witness.
• Aggregation/Analysis (Fillings): Where data from production is pulled in, scanned, and analyzed using tools like GUAC, Vista, OSV, and Security Insights.
• Consumption (Bottom Bun): Where consumers leverage the analyzed data to ask critical questions about the software (e.g., "Did it follow Salsa at what level? Is this compliant with internal policy?").
• Rules/Enforcement (Left Slice): The foundational principles and mechanisms like In-Toto, Sigstore, and SLSA Baseline that ensure the integrity and trustworthiness of the sandwich.

1. A Four-Pillar Approach to OSS Security is Essential: Lieberman outlined a progression for robust open-source security:

• Trust Foundation: Identifying who is involved in the supply chain.
• Attestation: Recording verifiable facts about the software's origin and build process.
• Aggregation & Synthesis: Combining attestations to understand the scope and impact of issues.
• Insight, Policy & Action: Applying intelligence to enforce policies and automate responses.

1. GUAC is a Centralizing Force: The Graph for Understanding Artifact Composition (GUAC), an incubating OpenSSF project, emerged as a key contribution. It functions as a graph database designed to ingest, parse, and correlate diverse supply chain security metadata (SBOMs, vulnerability data, license info, build attestations) into a unified, queryable graph. This enables powerful insights, policy checks, patch planning, and identification of critical infrastructure.

1. The Need for an "SDLC Control Plane": Looking ahead, Lieberman envisioned a future where an SDLC control plane or supply chain control plane manages the state across an entire ecosystem. Similar to how Kubernetes orchestrates workloads, this abstraction would control what is pulled into an ecosystem, ensuring it's built correctly, developed by approved parties, and adheres to established policies.

Technical Deep Dive

▶ Watch: Detailed breakdown of SDLC attack vectors (4:40)

The technical core of Lieberman's presentation revolved around the "Open Source Security Sandwich" and the underlying data and tools required to implement it. This framework provides a structured approach to a problem often perceived as insurmountable.

At the "Production" layer (the top bun of the sandwich), the focus is on generating high-quality, verifiable metadata about how software is built. This involves:

• Salsa (Software Supply Chain Levels for Artifacts): A framework developed by the OpenSSF that defines a set of progressive security levels for artifacts, focusing on provenance. It ensures that what is built truly originated from the specified source code. This is crucial for detecting build-time compromises like SolarWinds.
• VEX (Vulnerability Exploitability eXchange): While often associated with consumption, VEX plays a role in production by allowing producers to attest to the non-applicability or mitigation status of known vulnerabilities within their components.
• OpenSSF Scorecard: An automated tool that assesses various security health metrics for open-source projects, providing insights into their adherence to best practices during production.
• Allstar and Witness: Tools that help enforce security policies and attest to the integrity of build processes.

The "Aggregation and Synthesis" layer (the sandwich filling) is where the diverse metadata generated during production is collected, processed, and made actionable. This is where GUAC (Graph for Understanding Artifact Composition) shines. GUAC is an incubating OpenSSF project designed to:

• Ingest Diverse Metadata: It pulls in information from various sources, including SPDX and CycloneDX SBOMs, OpenSSF Scorecard results, OSV (Open Source Vulnerability) database entries, and license information from services like Clearly Defined and Depths.dev.
• Construct a Graph Database: GUAC transforms this disparate data into a unified graph of facts and information. Each node in the graph represents an artifact (package, source, build, vulnerability), and edges represent relationships (e.g., "this package depends on that package," "this package has this vulnerability," "this package was built with this Salsa level").
• Enable Powerful Queries: Using GraphQL, GUAC allows users to ask complex questions about their software supply chain. For example:
• "What are the dependencies of the Prometheus Golang client?" (demonstrated with a query showing a package URL, its known vulnerabilities, and source repository).
• "What software was built with a specific container digest?" (querying by hash to identify components within a Kubernetes image).
• "Which software has a specific Salsa attestation level?" (identifying components that meet certain provenance standards).
• Queries can also retrieve license information, vulnerability status, and even assist in patch planning by identifying all affected components.

The "Consumption" layer (the bottom bun) leverages the aggregated data from GUAC and other tools to inform decision-making. Consumers can query the graph to:

• Verify Salsa levels of ingested software.
• Check adherence to internal security policies.
• Assess the risk profile of new dependencies.
• Rapidly identify the blast radius of a newly discovered vulnerability (e.g., "Is this a one-off or a Log4j-like situation affecting hundreds of applications?").

The "Rules and Enforcement" layer (the left slice) represents the foundational mechanisms that ensure the integrity of the entire process:

• In-Toto: A framework for securing the integrity of software supply chains end-to-end by defining and verifying the steps and actors involved in the software's journey. It uses layout files to describe allowed actions.
• Sigstore: Provides a free, open-source service for signing and verifying software artifacts, ensuring that software originates from a trusted source and has not been tampered with.
• The Update Framework (TUF): A robust system for securing software updates, protecting against various attacks like rollback, freeze, and mix-and-match attacks, even when repositories are compromised.
• S2C2F (Secure Software Consumption Control Framework): Described as a set of rules for consuming software, complementing Salsa's focus on production. Lieberman noted it will soon be integrated into the Salsa project for a more holistic view.
• OCAL (Open Control Assessment Language): An XML-based format for expressing compliance and security control information, useful for managing control spreadsheets and assessments.

Lieberman emphasized that this is not just about individual tools but about an event-based system where changes in the ecosystem trigger analysis and policy enforcement. The ultimate vision is an SDLC control plane, an abstraction that manages the state of all ingested software, ensuring it meets defined standards for building, development, and project approval across the entire ecosystem.

Demo / Proof of Concept

▶ Watch: Artifact publication and distribution channel attack examples (6:40)

While the talk did not feature a live, interactive demonstration, Mike Lieberman effectively used several screenshots of GUAC (Graph for Understanding Artifact Composition) queries and their corresponding results to illustrate its capabilities. These served as a powerful proof of concept, showcasing how the tool translates theoretical concepts of supply chain metadata aggregation into practical, actionable insights.

The visual demonstrations included:

1. Dependencies of a Specific Package: A GraphQL query was shown to identify the dependencies of the Prometheus Golang client. The output clearly displayed information such as the package URL (e.g., pkg:golang/github.com/prometheus/[email protected]), its current vulnerability status (e.g., "no known vulnerability"), and its associated source code repository (e.g., git.github.com/prometheus/client_golang). This demonstrates GUAC's ability to quickly map out the direct relationships of a component.

1. Information by Container Digest: Another query illustrated how to retrieve information based on a container digest (a cryptographic hash of a container image). For instance, a query related to a Kubernetes image's hash (sha256:0d6529...) revealed the software components within that specific build, including details about their origin and build processes. This highlights GUAC's utility in connecting low-level artifact identifiers to higher-level software metadata.

1. Software with Salsa Attestations: A crucial demonstration involved querying for software that possesses Salsa (Software Supply Chain Levels for Artifacts) attestations. This capability allows organizations to verify which components in their supply chain have been built with a verifiable provenance, indicating a higher level of security assurance. The query output showed specific software artifacts and their associated Salsa levels, enabling policy enforcement based on desired security postures.

Beyond these specific examples, Lieberman mentioned GUAC's ability to answer a wide range of questions, including:

• Retrieving license information for all components.
• Accessing comprehensive vulnerability information from various sources.
• Assisting in patch planning by identifying all instances of a vulnerable component across the dependency graph.

These visual examples effectively conveyed GUAC's role as a centralized intelligence hub for software supply chain data. They demonstrated how a graph database approach, combined with open standards like SBOMs and Salsa, can provide unprecedented observability and enable data-driven decision-making for security teams.

Defensive Implications

▶ Watch: “Turtles all the way down”: recursive supply chain problem (8:00)

The detailed analysis of software supply chain threats and the proposed "Open Source Security Sandwich" framework offer clear and actionable defensive implications for organizations:

1. Adopt a Threat Modeling Mindset: Defenders must proactively threat model their entire SDLC, not just runtime environments. This involves identifying critical assets, understanding potential risks at each stage (producer, SCM, build, distribution, consumption), and designing appropriate controls. The security approach should be proportionate to the risk profile; a personal blog will have different requirements than an online banking application, necessitating tailored investments in time, resources, and tools.

1. Embrace Open Specifications and Tools: Move away from proprietary solutions towards open standards for security metadata.

• Mandate SBOMs: Generate and consume Software Bill of Materials (SBOMs) (SPDX, CycloneDX) for all software, recognizing them as a foundational, machine-readable inventory. While tooling is maturing, their adoption is critical for visibility.
• Utilize Provenance Standards: Implement Salsa to attest to how software is built, ensuring source code integrity and build process verification.
• Leverage VEX: Use Vulnerability Exploitability eXchange (VEX) to communicate the actual impact of vulnerabilities within your specific context, reducing alert fatigue.
• Adopt Signing and Update Frameworks: Implement Sigstore for signing artifacts and The Update Framework (TUF) for securing software updates to guarantee authenticity and prevent tampering.

1. Prioritize Observability and Data Aggregation:

• Track Everything: Implement mechanisms to track software throughout its entire lifecycle, from developer commits to deployment. This includes collecting metadata on network traffic, build processes, and artifact publications.
• Centralize with a Graph Database: Utilize tools like GUAC to aggregate and synthesize diverse supply chain security metadata (SBOMs, vulnerability reports, license info, build attestations). This creates a unified, queryable source of truth, enabling comprehensive analysis and reducing the "unknown unknowns."

1. Implement Layered Controls and Defense-in-Depth:

• Runtime Controls: Maintain traditional runtime security measures (firewalls, IDS/IPS, WAFs) as a last line of defense.
• Dependency Management: Implement rigorous processes for ingesting dependencies, including third-party risk management, open-source risk management, and automated scanning (SAST, DAST).
• Policy Enforcement: Define clear security policies and use tools to enforce them throughout the SDLC. Leverage aggregated data to automatically raise alarms, remediate issues, or prevent non-compliant software from entering production.

1. Engage with the Open-Source Security Community: Actively participate in and contribute to initiatives by organizations like the OpenSSF and CNCF. These communities are developing the standards and tools necessary to secure the collective software supply chain. Contributing to projects like GUAC or Salsa directly strengthens the ecosystem that all organizations rely upon.

1. Build an "SDLC Control Plane": While a long-term vision, organizations should strive towards conceptualizing and building capabilities for an SDLC control plane. This involves managing the state of all software artifacts, enforcing policies on how they are built and consumed, and ensuring only approved components enter the ecosystem. This abstraction provides comprehensive oversight and governance across the entire software supply chain.

By adopting these defensive strategies, organizations can transition from a reactive, vulnerability-patching approach to a proactive, data-driven, and resilient software supply chain security posture, significantly reducing their exposure to sophisticated attacks.

Key Takeaways

• Holistic Security is Paramount: Software supply chain security encompasses the entire SDLC, from the production of software to its consumption, requiring a comprehensive, layered approach as illustrated by the "Open Source Security Sandwich" model.
• Data and Observability are Foundational: Effective supply chain security relies on generating, collecting, and analyzing explicit metadata about software artifacts and processes. Without knowing "who did what and when," identifying and remediating compromises is nearly impossible.
• Open Standards Drive Interoperability: Adopting open specifications like SBOMs (SPDX, CycloneDX), Salsa for provenance, and VEX for vulnerability exploitability is crucial for standardized communication, machine readability, and broad community support.
• GUAC Centralizes Supply Chain Intelligence: The Graph for Understanding Artifact Composition (GUAC) serves as a powerful graph database for aggregating diverse metadata from various sources, enabling sophisticated queries and providing actionable insights for policy enforcement and risk management.
• Proactive Defense Requires Layered Controls: Defenders must implement a multi-faceted strategy covering threat modeling, rigorous dependency management, artifact signing (Sigstore), secure updates (TUF), and automated policy enforcement across the entire supply chain.
• Community Engagement is Essential: Participating in and contributing to open-source security initiatives like those from the OpenSSF and CNCF strengthens the collective security posture and accelerates the development of critical tools and standards.

About the Speaker(s)

Mike Lieberman is the Co-founder and CTO of Casari, a company dedicated to software supply chain security. His extensive background includes significant work in fintech and banking prior to his current role. A recognized authority in the field, Lieberman is a co-author of the book "Securing the Software Supply Chain." He is deeply involved in the open-source community, serving as an OpenSSF Technical Advisory Council member and a Governing Board member. Additionally, he is a CNCF TAG Security Tech Lead and a maintainer of several influential open-source projects, including GUAC and Salsa, which are instrumental in advancing software supply chain security.