Attesting and Verifying Your Software Supply-Chain With In-toto - Alan Chung Ma & Justin Cappos

Attesting and Verifying Your Software Supply-Chain With In-toto

Speakers: Alan Chung Ma, Software Engineer; Justin Cappos, Professor, NYU

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Cydz0hadVuQ

Overview

In an era of escalating software supply chain attacks, securing the integrity and provenance of software artifacts has become paramount. This talk by Justin Cappos, a professor at NYU and creator of In-toto, and Alan Chung Ma, an In-toto maintainer, delves into how In-toto provides a robust framework for attesting and verifying the software supply chain. They highlight In-toto's role in establishing transparency and cryptographic guarantees throughout the software development lifecycle, from source control to distribution.

The presentation addresses the critical need for verifiable evidence in software production, driven by both the increasing sophistication of attacks and stringent regulatory compliance mandates. By offering a standardized approach to generating and consuming attestations, In-toto empowers organizations to cryptographically prove that their software was built and handled according to defined policies, effectively bridging the gap between security best practices and compliance requirements.

This article will explore the fundamental concepts of software supply chain security, detail In-toto's technical architecture, introduce key tools that facilitate its adoption, and discuss its profound implications for both defensive strategies and regulatory adherence.

Background

▶ Watch: Introduction to IntoTo and software supply chain security (0:00)

The software supply chain encompasses the entire process by which software is created, from initial code commits and dependency acquisition to compilation, testing, packaging, and final distribution. This intricate web involves human developers, automated systems, third-party libraries, and various infrastructure components. A software supply chain attack occurs when a malicious party (or even an accidental misstep) compromises any stage of this process, leading to unexpected or undesirable behavior in the final product.

The speakers underscored the alarming rise in such incidents, noting an "exponential growth" in attacks since the early 2000s. Initially, these sophisticated attacks were primarily the domain of nation-state actors, targeting critical infrastructure like power plants or financial systems. However, they have increasingly become mainstream, affecting common package repositories and a wider array of companies.

Several high-profile incidents illustrate the diverse vectors of these attacks:

• Version Control System Compromises: Allegations of the NSA embedding a backdoor into Juniper products, affecting VPN connections for an extended period.
• Build System Infiltrations: The notorious SolarWinds attack, where Russian hackers allegedly inserted backdoors into updates, compromising numerous Fortune 500 companies and US government entities.
• Packaging and Distribution Manipulation: The XcodeGhost incident, where a malicious version of Apple's Xcode IDE was distributed via unofficial mirrors in China, leading developers to unknowingly include malware in their applications.
• Testing Failures: Microsoft accidentally pushing a sensitive update to mirrors serving repressive regimes and later repeating the mistake, raising suspicions about targeted surveillance. Similarly, Apple and others have inadvertently deployed beta versions to production.

These incidents underscore the pervasive vulnerability of traditional software supply chains. In response to this growing threat, governments and regulatory bodies worldwide have begun mandating stricter security controls. Notable examples include:

• US Executive Order on Improving the Nation’s Cybersecurity: Mandating the use of Software Bill of Materials (SBOMs), which list all components in a piece of software.
• European Cyber Resilience Act: Further mandating specific controls and protections for digital products.
• NIST Secure Software Development Framework (SSDF): Providing guidance for secure software development practices.
• Similar initiatives in the UK, Japan, and other countries.

These compliance efforts, while seemingly bureaucratic, are presented by the speakers as a critical "enabler" for better security. By requiring verifiable evidence and transparency, these regulations push organizations towards adopting mechanisms like In-toto attestations, which can serve both compliance needs and enhance overall security posture.

Key Findings

▶ Watch: Understanding what constitutes a software supply chain attack (1:48)

The core premise of In-toto is to achieve compliance with transparency by collecting and verifying evidence throughout the software supply chain. The talk highlighted several key findings and contributions that collectively enable this goal:

1. Standardized Attestation Framework: In-toto provides a common language for diverse tools and processes to generate security-relevant information. This standardization ensures that attestations from GitHub, CI/CD pipelines, C-SLSA builds, or Debian packaging all adhere to a consistent format, making them universally understandable and verifiable.
2. Cryptographically Verifiable Evidence: At the heart of In-toto is the concept of attestations – cryptographically signed statements about specific actions or properties within the supply chain. These attestations act as tamper-proof records, ensuring that the evidence collected is trustworthy and hasn't been altered.
3. Comprehensive Attestation Data: In-toto attestations are designed to be flexible, capturing a wide array of evidence. They consist of subjects (the artifacts being attested to, e.g., a DBN file) and predicates (the evidence itself). Predicates can include SPDX (for SBOMs), Links (detailing inputs, outputs, and commands of a step), SLSA Provenance (capturing build details, machine state, and versions), as well as information about releases, runtime traces, test results, and vulnerabilities.
4. Practical Tooling for Implementation: The In-toto project provides or integrates with several essential tools to facilitate its adoption:

• In-toto Witness: A production-ready implementation for generating attestations, designed to be easily integrated into existing CI/CD contexts like GitHub Actions or GitLab CI.
• Archivist: A specialized storage solution for In-toto attestations, enabling efficient querying and maintaining relationships between attested subjects.
• GUAC (Graph for Understanding Artifact Compositions): A powerful visualization tool that consumes attestations, SBOMs, and vulnerability information to map out complex supply chain relationships, identify gaps, and uncover potential threats.

1. Policy Enforcement through Layouts: In-toto's layouts define the expected "shape" of a secure supply chain, specifying the sequence of steps, the inputs (materials) and outputs (products) of each step, and the trusted functionaries (identities) responsible for each action. These layouts, cryptographically signed by the project owner, enable robust policy enforcement, ensuring that software artifacts adhere to predefined security postures.
2. Community Adoption and Evolution: The project has seen significant adoption across the industry, with integrations into GitHub Actions (for build provenance), Homebrew (for package bottling attestations), and PyPI (for package uploads). In-toto has also successfully graduated to a CNCF incubation project, signifying its maturity and widespread impact. Ongoing efforts, particularly within the In-toto Policy Working Group, are focused on enhancing policy enforcement to fully leverage the richness of modern attestations and explore policy engine-agnostic approaches.

Technical Deep Dive

▶ Watch: High-profile examples of software supply chain compromises (2:05)

In-toto's technical architecture is built upon the principle of verifiable transparency, establishing a chain of trust from the earliest stages of software development to its final deployment. The core components facilitating this are its attestation framework, specialized tooling, and policy enforcement mechanisms.

At the foundation is the In-toto Attestation Framework. An attestation is essentially a digital envelope containing cryptographically signed metadata about a specific event or artifact in the supply chain. Each attestation comprises two primary elements:

• Subjects: These are the artifacts being attested to, typically identified by their cryptographic hash. For example, if a Debian package is being attested, the package file itself would be a subject.
• Predicate: This is the actual evidence or data describing the action taken or the properties of the subject. In-toto supports a variety of predicate types to cater to different supply chain activities:
• SPDX (Software Package Data Exchange): Used to embed SBOMs, providing a comprehensive list of components, licenses, and copyrights.
• Links: A foundational predicate type that captures the inputs (materials), outputs (products), and the exact command executed for a specific step. This provides a direct record of how an artifact was transformed.
• SLSA Provenance: A critical predicate for build integrity, detailing information such as the specific machine on which a build was run, its version, and even its current state. This enables higher levels of confidence in the build process.
• Other predicate types can cover releases, runtime traces, test results, and vulnerability information, offering a versatile mechanism to capture any relevant security metadata.

To facilitate the generation of these attestations, In-toto offers In-toto Witness. Developed by TestifySec and donated to the In-toto project, Witness is a production-ready command-line interface (CLI) that integrates seamlessly into existing CI/CD pipelines (e.g., GitHub Actions, GitLab CI). It operates by observing a specific context, capturing the input files, output files, and the commands executed within a particular step. Witness then pulls a signing key from a secure key provider (such as a cloud provider's Key Management Service or a Hardware Security Module) to cryptographically sign the generated attestation. This automation replaces manual compliance checks with verifiable, machine-generated evidence.

Once attestations are generated, they need to be stored and managed efficiently. Archivist, another tool from TestifySec, addresses this need. It provides a centralized repository for In-toto attestations, maintaining relationships between subjects and enabling powerful querying capabilities. Users can retrieve attestations based on specific criteria, such as the person or machine that generated them, or the type of attestation (e.g., all SLSA provenance attestations). This structured storage is vital for auditability and compliance reporting.

Understanding the complex interdependencies within a software supply chain can be challenging with raw attestations. This is where GUAC (Graph for Understanding Artifact Compositions) becomes invaluable. GUAC is a visualization and analysis tool that ingests not only In-toto attestations but also SBOMs and vulnerability information. It constructs a comprehensive graph of relationships between artifacts, dependencies, and their associated metadata. While not strictly essential for In-toto's core functionality, GUAC helps users visualize their supply chain, identify gaps in attestation coverage, uncover potential threats, and establish clear relationships between components. The speakers emphasized that even though supply chains can generate "really big graphs," GUAC makes them digestible.

For enforcing specific security policies, In-toto utilizes Layouts. An In-toto layout defines the expected sequence of steps in a supply chain, effectively describing its "shape." For each step (e.g., source control, test, build, package), the layout specifies:

• Materials: The expected inputs to the step.
• Products: The expected outputs generated by the step.
• Trusted Functionaries: The cryptographic identities (keys, SPIFFE identities, ephemeral Sigstore keys, etc.) authorized to perform that step.
• Relationships: The links between materials and products across different steps, ensuring that the outputs of one step correctly serve as inputs for the next.

The entire layout is cryptographically signed by the project owner, making it a tamper-proof blueprint for the supply chain. When a software artifact is built, In-toto verifies that the generated attestations conform to the signed layout, ensuring that the process adhered to the defined policies.

The speakers noted that earlier In-toto layouts primarily relied on the more limited "Links" predicate. However, with the advent of richer attestations like SLSA Provenance, the community recognized the need to evolve policy enforcement. The In-toto Policy Working Group is actively addressing this, with efforts like the In-toto Attestation Verifier prototype bridging the gap to leverage richer attestation data in layouts. The group is also exploring broader goals, such as improving usability, defining more flexible policies, and designing policy engine-agnostic approaches (e.g., exploring integration with Macaroon) to enable policy sharing across teams and organizations.

The relationship between In-toto and other prominent supply chain security projects, Sigstore and SLSA, was also clarified. In-toto attestations are a major driving force behind Sigstore's Rekor transparency log, with hundreds of thousands of In-toto attestations being stored there. One of In-toto's creators, Santiago Torres Aaras, is also a creator of Sigstore, highlighting the inherent synergy. Furthermore, SLSA (Supply-chain Levels for Software Artifacts) initially originated as a sub-project of In-toto. A SLSA attestation is, in essence, an In-toto attestation with a predicate specifically formatted to meet SLSA's opinionated requirements for source, build, and dependency integrity levels. In-toto acts as the underlying "signing layer" or "transport protocol," providing cryptographic compatibility, while SLSA defines the specific policy boundaries and compliance levels.

Demo / Proof of Concept

▶ Watch: The alarming exponential growth of supply chain attacks (4:53)

While the talk did not feature a live demonstration or a hands-on proof of concept, the speakers thoroughly detailed the functionalities and integration points of key In-toto tools. They described how In-toto Witness operates as a production-ready implementation, capable of generating cryptographically signed attestations within existing CI/CD environments like GitHub Actions or GitLab CI. This process involves capturing inputs, outputs, and commands, then signing the resulting attestation using a key from a secure provider.

The presentation also outlined the practical application of Archivist for storing and managing these attestations, emphasizing its ability to maintain relationships between subjects and enable efficient querying. Finally, the role of GUAC was illustrated conceptually with an example graph, showing how it visualizes artifact dependencies and relationships by consuming attestations, SBOMs, and vulnerability data. These descriptions collectively painted a clear picture of how In-toto can be implemented and leveraged in real-world software supply chains.

Defensive Implications

▶ Watch: Overview of compliance and regulations like SBOMs and CRA (6:17)

For security practitioners and organizations, In-toto offers a powerful and comprehensive framework to significantly bolster software supply chain defenses and meet evolving regulatory requirements. The defensive implications are multi-faceted:

1. Establish Verifiable Provenance: By integrating In-toto Witness into every step of the CI/CD pipeline, organizations can automatically generate cryptographically signed attestations for source code, build processes, test results, and final packages. This creates an auditable, tamper-proof record of every transformation and action, effectively eliminating "blind spots" in the supply chain.
2. Enforce Granular Security Policies: In-toto layouts enable the definition and enforcement of strict policies regarding the sequence of operations, the expected inputs and outputs, and the specific trusted identities (functionaries) authorized to perform each step. This allows defenders to mandate controls such as "two-person sign-off" for critical stages or ensure that builds only occur on approved, hardened infrastructure. The evolution towards leveraging richer attestation data will enable even more precise policy enforcement.
3. Enhance Auditability and Compliance: The ability to generate and store standardized, cryptographically verifiable attestations is crucial for demonstrating compliance with mandates like SBOMs (US Executive Order), the European Cyber Resilience Act, and NIST SSDF. Tools like Archivist facilitate the centralized storage and efficient retrieval of this evidence, streamlining audits and providing concrete proof of adherence to secure development practices.
4. Proactive Threat Identification and Remediation: GUAC allows security teams to visualize the entire software supply chain, identifying unexpected dependencies, missing attestations, or deviations from the intended layout. This "single pane of glass" view helps uncover potential attack vectors, expose gaps in security controls, and understand the impact radius of compromised components more effectively.
5. Integrate with a Broader Security Ecosystem: In-toto is designed to work synergistically with other critical security projects. Its attestations are a primary input for Sigstore's Rekor transparency log, providing an immutable public record of software artifacts. Its relationship with SLSA allows organizations to define and verify specific levels of supply chain integrity, aligning with industry best practices for secure software development.
6. Mitigate Against Insider Threats and Accidental Compromises: By requiring cryptographic signatures from trusted functionaries and verifying against a signed layout, In-toto reduces the risk of unauthorized modifications, whether from malicious insiders or accidental misconfigurations. Any deviation from the defined process will trigger a verification failure.
7. Shift Left in Security: By embedding attestation generation and verification throughout the development process, In-toto promotes a "shift left" approach to security. Issues related to provenance and integrity can be identified and addressed much earlier, reducing the cost and complexity of remediation later in the lifecycle.

It is important to note that while In-toto provides robust mechanisms for verification, it does not absolve developers of their responsibilities. As the speakers highlighted, In-toto can confirm that a specific compiler version was used, but it cannot guarantee that the compiler itself is bug-free or that code reviews were conducted diligently. Defenders must therefore combine In-toto's technical controls with strong security hygiene, developer training, and a culture of continuous vigilance.

Key Takeaways

• In-toto enables verifiable software supply chain transparency: It provides a framework for generating and verifying cryptographically signed attestations at every step of the software development lifecycle.
• Addresses growing supply chain attacks and compliance needs: In-toto helps organizations combat the increasing threat of supply chain compromises and meet stringent regulatory mandates like SBOMs and the European Cyber Resilience Act.
• Standardized attestations with rich metadata: The framework defines a common format for attestations, encompassing subjects (artifacts) and predicates (evidence like SPDX, SLSA Provenance, or custom data), ensuring universal interpretability.
• Practical tools facilitate adoption: Tools like In-toto Witness automate attestation generation, Archivist provides secure storage and querying, and GUAC offers powerful visualization of supply chain relationships.
• Policy enforcement through signed layouts: In-toto layouts define the expected structure, trusted functionaries, and relationships within a supply chain, allowing for robust, verifiable policy enforcement.
• Foundational to the security ecosystem: In-toto attestations are widely adopted by projects like Sigstore's Rekor and serve as the underlying format for SLSA provenance, demonstrating its critical role in the broader software supply chain security landscape.

About the Speaker(s)

Justin Cappos is a professor at NYU and one of the original creators of In-toto. His extensive background in systems security and his pioneering work on In-toto underscore his commitment to advancing software supply chain security. He also guided Santiago Torres Aaras, a PhD student of his, who went on to become one of the creators of Sigstore, highlighting the deep technical and intellectual ties between these critical security projects.

Alan Chung Ma is a software engineer currently working at a small company and serves as a maintainer of the In-toto project. His role as a maintainer reflects his direct involvement in the ongoing development and evolution of the In-toto framework, ensuring its continued relevance and robustness for the community.