Practical (and impractical) git commit signing

Practical (and impractical) git commit signing

Speakers: Matthew Garrett

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=w9SN_15gUhc

Overview

In this insightful talk, Matthew Garrett delves into the complexities and practicalities of Git commit signing, a crucial but often misunderstood aspect of software supply chain security. Garrett, a former Linux kernel developer with a keen interest in hardware-backed cryptographic keys, critically examines the existing methods for signing Git commits – GPG, X.509 certificates, and SSH keys – highlighting their inherent flaws and limitations, particularly in enterprise environments. The core of his presentation advocates for a paradigm shift towards SSH certificates as the most viable and secure solution for robust commit signing, offering unparalleled flexibility, manageability, and the potential for hardware attestation.

Garrett argues that while no single solution can "fix" supply chain security entirely, effective commit signing can significantly reduce risks by enhancing the provenance and trustworthiness of code. By demonstrating possession of a private key, a signature provides evidence that a commit originated from an authorized source, making it harder for attackers to inject malicious code or impersonate legitimate developers. The talk is particularly relevant for organizations struggling with secure developer workflows, those looking to implement stronger controls over their codebases, and anyone interested in the practical application of cryptography in software development.

Background

▶ Watch: Speaker introduction, background, and foreshadowing hardware keys (0:00)

The necessity for Git commit signing stems from the fundamental need to establish code provenance and enhance supply chain security. In an era of escalating software supply chain attacks, ensuring the authenticity and integrity of every line of code is paramount. Git, as the de facto standard for version control, offers mechanisms to sign commits and tags, providing cryptographic assurance that the content was created or approved by a specific entity possessing a corresponding private key. This evidence helps mitigate risks such as unauthorized code injection, impersonation of developers, and the compromise of developer accounts.

Git supports several cryptographic methods for signing: GPG (GNU Privacy Guard), X.509 identity certificates (commonly used for S/MIME), and SSH keys. While Git also supports signing tags, Garrett emphasizes the importance of signing individual commits. Tag signing, though useful for marking releases, only asserts that the merger approved the collection of commits, not that each individual commit's author is verified. Focusing on commit signing aims to make it a ubiquitous and transparent part of the development workflow. Garrett's underlying enthusiasm, as he notes, lies in the secure storage of cryptographic keys within hardware elements that are not directly accessible by the operating system, foreshadowing his preferred solutions.

However, each of Git's native signing methods presents its own set of challenges, particularly for organizations aiming for scalable and auditable security. The speaker critically evaluates these methods, revealing fundamental design flaws or practical impediments that hinder their effective adoption for enterprise-grade supply chain security.

Key Findings

▶ Watch: Setting expectations: not a supply chain security fix (3:00)

Garrett systematically dissects the shortcomings of traditional Git commit signing methods and presents a compelling case for SSH certificates as the superior alternative:

1. GPG's Inherent Flaws: GPG, despite being the oldest and most integrated signing method in Git, is severely criticized. Its web of trust model is deemed outdated and impractical for modern organizational contexts, as it relies on a decentralized trust network that rarely maps to legal or online identities in a meaningful way. For enterprises, managing GPG keys, especially short-lived ones, and associating them with hardware attestation is "painful and miserable," lacking the necessary infrastructure for centralized control and revocation. The speaker openly admits his "hatred" for GPG, underscoring its practical difficulties.

1. X.509's Complexity and Trust Issues: X.509 certificates, while offering a more structured PKI, are plagued by complexity (especially with ASN.1 encoding) and practical limitations. The primary assertion from readily available S/MIME certificates is often just email address verification, which may not be sufficient for robust identity. Furthermore, platforms like GitHub and GitLab typically only trust X.509 certificates issued by globally trusted Certificate Authorities (CAs), preventing organizations from using internal CAs for commit signing. This creates hurdles for managing short-lived certificates and handling revocations efficiently.

1. GitHub's Inadequate Verification Model: A significant finding is GitHub's current approach to commit verification. For GPG and SSH keys, GitHub's "Verified" status relies on users adding their public keys to their accounts. This model is vulnerable: if a user's GitHub account is compromised, an attacker can simply add a new key, sign malicious commits, and have them appear "Verified." While MFA can protect against adding keys, a sufficiently compromised account can bypass this. For X.509, it relies on globally trusted CAs and email address matching. Crucially, even GitHub Enterprise, which allows organizations to enforce SSH certificate-based authentication, lacks the corresponding feature to verify commit signatures against the same organizational CA.

1. SSH Certificates as the Optimal Solution: Garrett champions OpenSSH certificates as the most practical and powerful solution. Unlike GPG and X.509, SSH certificates make "no pretense of identity" beyond being a key pair, but they allow for the embedding of arbitrary signed metadata. This metadata is key to their utility:

• Simplicity: No complex intermediates or ASN.1 encoding. Certificates are directly signed by a CA, simplifying PKI management.
• Flexibility: Organizations can be their own CA, issuing certificates with custom validity periods (e.g., short-lived, 24-hour certificates) and rich metadata (e.g., group memberships, permissible repository paths). This dramatically simplifies revocation management.
• Usability: SSH agent forwarding allows local keys to be used for signing on remote systems without special configuration, enhancing developer experience.
• Hardware Backing: SSH certificates seamlessly integrate with hardware-backed key stores like Trusted Platform Modules (TPM) and the Apple Secure Enclave, enabling strong device identity attestation. This allows organizations to assert that a commit was signed on a trusted, managed device.

In essence, the talk reveals that while Git offers signing capabilities, the existing implementations (GPG, X.509) are often impractical or insufficient for robust enterprise security. SSH certificates, by contrast, provide the necessary cryptographic primitives, flexibility, and integration points to build a truly secure and manageable commit signing infrastructure.

Technical Deep Dive

▶ Watch: Overview of Git's supported signing methods (4:45)

The technical core of Garrett's argument lies in comparing the underlying mechanisms and trust models of GPG, X.509, and SSH certificates, ultimately demonstrating why the latter is superior for enterprise-level Git commit signing.

GPG (GNU Privacy Guard):

GPG's signing mechanism is rooted in its web of trust model. This decentralized trust network relies on individuals signing each other's public keys to attest to identity. While theoretically robust, in practice, this model "went out of fashion in the early 2000s" and rarely aligns with modern organizational identity management. For an enterprise, establishing trust in GPG keys is arduous:

• Identity Assertion: A GPG key can claim any identity, and verifying this identity requires a cumbersome process, often involving physical presence or complex out-of-band verification.
• Key Management: Organizations would need to build custom infrastructure to manage employee GPG keys, including a root key to sign employee keys. This is "immensely tedious" for onboarding, offboarding, and role changes.
• Short-lived Keys: GPG's design makes it difficult to implement short-lived keys or certificates, which are crucial for mitigating the impact of key compromise and simplifying revocation.
• Hardware Attestation: Making an assertion that a GPG key was generated on a specific, trusted piece of hardware is "very difficult," limiting its use in device-attested security models.

X.509 Identity Certificates:

X.509 certificates, as used in S/MIME, offer a more hierarchical Public Key Infrastructure (PKI) model. While the CA/Browser Forum has attempted to standardize baseline parameters, adoption outside of browsers for email signing is inconsistent.

• Trust Model: The practical trust model for easily obtainable S/MIME certificates often only verifies possession of an email address. While this can be "appropriate model" for community projects, it's insufficient for strong identity in an enterprise.
• Platform Limitations: Platforms like GitHub and GitLab impose a critical constraint: they "will only trust X.509 certificates that are issued by a globally trusted certificate authority." This means organizations cannot use their own internal CAs for commit signing verification on these platforms, forcing reliance on third-party CAs and making custom policy enforcement impossible.
• Revocation and Short-lived Certificates: The reliance on external CAs complicates revocation and the issuance of short-lived certificates. Requesting and verifying a new X.509 certificate "every 24 hours" for each user is not feasible, leading to longer-lived certificates that pose greater security risks if compromised.
• Complexity: X.509's underlying ASN.1 encoding and complex RFCs contribute to its general unpopularity and difficulty in implementation.

SSH Certificates (The Recommended Approach):

Garrett positions SSH certificates, specifically as defined by OpenSSH, as the most elegant and practical solution. They fundamentally differ from GPG and X.509 by making "no pretense of identity" beyond being a cryptographic key pair. The power lies in their ability to embed signed metadata.

• Structure: An SSH certificate is "just an SSH key type with an embedded public key and a bunch of signed metadata wrapping that signed public key." This simple structure avoids the complexity of ASN.1 and intermediate CAs.
• CA Model: SSH certificates are directly signed by a Certificate Authority (CA). This CA can be "whoever you want it to be," meaning an organization can run its own internal SSH CA, gaining full control over its trust anchors.
• Metadata Flexibility: The embedded metadata is highly customizable. Organizations can include:
• User identity: Linking the certificate to an internal user ID.
• Group memberships: For role-based access control.
• Authorization constraints: "This user should only be able to commit to these paths in my mono repo or these repositories in my organization." This enables fine-grained authorization policies directly within the certificate.
• Short-lived Certificates: Issuing short-lived certificates (e.g., expiring every 24 hours) is straightforward. This dramatically simplifies revocation management, as compromised keys have a very limited window of utility.
• Usability with SSH Agent: SSH agent forwarding (using ssh -A) allows developers to sign commits on remote systems (e.g., dev environments, CI/CD pipelines) using their local hardware-backed keys without exposing the private key to the remote system. This is a "huge usability advantage."
• Git Integration: Git's signing operation, "horrifyingly" (according to Garrett) shells out to an external command. For SSH keys, this is ssh-keygen. Despite its name, ssh-keygen can perform signing operations, making SSH certificate integration seamless once configured.
• Hardware Backing: This is a critical advantage. SSH certificates can be generated and managed by hardware-backed key stores such as Trusted Platform Modules (TPM) or the Apple Secure Enclave. While PKCS11 can technically support this for other key types, Garrett expresses strong disdain for it ("I really hate PKCS11"). The ability to combine SSH certificates with device identity attestation allows organizations to assert that a commit "was signed on hardware that my organization owns," ensuring compliance and enhancing confidence in code origin.

The contrast is stark: GPG and X.509 are either too rigid, too complex, or too dependent on external, untrustworthy CAs. SSH certificates, by contrast, offer a flexible, simple, and powerful framework that aligns perfectly with enterprise security requirements for managing identity, authorization, and hardware attestation in code signing.

Demo / Proof of Concept

▶ Watch: Problems with GPG's web of trust model (6:17)

While Matthew Garrett's presentation did not include a live, interactive demonstration, he explicitly refers attendees to external resources where a detailed write-up and the relevant code are publicly available. He states, "this URL contains both a more detailed write-up of everything... But the code is available at that URL. The write-up is there and there are links to public Git library pose that contain all the source code."

This approach emphasizes the practical, open-source nature of his proposed solution. The provided resources include instructions on how to configure Git to use SSH certificates for auto-signing, how to manage certificate issuance, and how to integrate hardware-backed keys. The speaker highlights that once configured, the process becomes transparent to the developer: "you do Git commit... and if you have auto signing turned on, it will just be signed. If you do commit amend, if you do a rebase, it will resign everything in the process." He also mentions an implementation of an SSH agent that supports hardware-backed keys is linked in his material. This allows interested parties to immediately explore and implement the concepts discussed in the talk.

Defensive Implications

▶ Watch: Why GPG is impractical for organizational use (8:00)

The insights from Matthew Garrett's talk provide clear, actionable strategies for organizations to significantly bolster their Git commit signing practices and, by extension, their software supply chain security:

1. Transition to SSH Certificates for Commit Signing: Organizations should move away from GPG and X.509 for Git commit signing due to their inherent complexities and security limitations. Embracing SSH certificates offers a more manageable, flexible, and robust solution for identity and authorization.

1. Implement Hardware-Backed Key Storage: To achieve the highest level of assurance, integrate hardware-backed keys for signing. Utilizing Trusted Platform Modules (TPM) on Windows/Linux or the Apple Secure Enclave on macOS ensures that private keys never leave a secure hardware boundary and enables device identity attestation. This allows organizations to verify that commits originate from managed and trusted devices.

1. Establish an Internal SSH Certificate Authority (CA): Organizations should operate their own SSH CA to issue certificates to developers. This provides complete control over the trust chain, certificate validity periods, and embedded metadata, eliminating reliance on external, globally trusted CAs that often impose unsuitable policies.

1. Leverage Short-Lived Certificates: Implement a policy of issuing short-lived SSH certificates (e.g., with a 24-hour expiry). This dramatically reduces the window of exposure if a private key is compromised, effectively simplifying revocation management and enhancing overall security posture.

1. Utilize SSH Certificate Metadata for Granular Access Control: Embed arbitrary metadata within SSH certificates to enforce fine-grained authorization policies. This can include group memberships, permissible repository paths, or even specific commit types, ensuring that developers can only sign commits for authorized contexts.

1. Implement Custom CI/CD Verification for SSH Signatures: As platforms like GitHub currently lack robust support for verifying commit signatures against an organization's internal SSH CA, defenders must build custom verification workflows within their CI/CD pipelines. A GitHub Action or similar mechanism can be deployed to inspect every commit in a Pull Request, verify its SSH signature against the organization's trusted CA key, and enforce policies based on the certificate's embedded metadata.

1. Educate Developers on Secure SSH Key Management: While SSH certificates simplify many aspects, developers still need to understand the importance of protecting their underlying SSH private keys and how SSH agent forwarding works for seamless, secure signing.

1. Advocate for Platform Enhancements: For users of GitHub Enterprise and similar platforms, it's crucial to advocate for native support for organizational SSH CA-based commit signature verification. As Garrett points out, it's "especially infuriating" that GitHub Enterprise supports SSH certs for authentication but not for commit signature verification.

By adopting these defensive strategies, organizations can move beyond merely "signed" commits to "verifiably trusted" commits, significantly improving their ability to detect and prevent supply chain attacks.

Key Takeaways

• Git commit signing is a critical component of supply chain security, aiming to reduce risk by proving code provenance, rather than offering a complete fix.
• Traditional signing methods like GPG and X.509 certificates present significant practical challenges for enterprise adoption due to their outdated trust models, management overhead, and limitations with short-lived certificates and hardware attestation.
• SSH certificates, particularly those defined by OpenSSH, offer a superior and highly flexible solution for Git commit signing, allowing organizations to define their own CAs, embed rich metadata, and issue short-lived certificates.
• Integrating hardware-backed keys (e.g., TPM, Apple Secure Enclave) with SSH certificates enables strong device identity attestation, ensuring commits originate from trusted and managed hardware.
• Current platform integrations, such as GitHub's "Verified" status, are often insufficient for organizational security needs, necessitating the implementation of custom CI/CD pipeline checks to verify SSH commit signatures against an organization's internal CA.
• The combination of short-lived SSH certificates, arbitrary metadata, and SSH agent forwarding provides a user-friendly, secure, and highly controllable framework for enterprise-grade Git commit signing.

About the Speaker(s)

Matthew Garrett is a distinguished security researcher and former Linux kernel developer known for his expertise in open-source software and hardware security. Despite humorously claiming "no formal qualifications in computers" beyond an 11-year-old's word processor certificate (his training is in fruit fly genetics), Garrett has an impressive track record. He has previously worked as a Linux kernel developer, occasionally "stole cos players one" (a humorous reference to winning awards), and has a history of "poking at various embedded devices," including famously breaking the security of rental electric scooters to enable free rides. He is particularly enthusiastic about the secure storage of cryptographic keys in hardware elements inaccessible by the operating system, a passion that underpins his advocacy for SSH certificates and hardware-backed solutions in this talk.