Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors

Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors

Speakers: Taylor R. Schorlemmer, Kelechi G. Kalu, Luke Chigges, Kyung Myung Ko, Eman Abdul-Muhd Abu Ishgair, Saurabh Bagchi

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=aja-VC4hJ2g

Overview

This technical article delves into the findings presented by Taylor R. Schorlemmer and co-authors at the IEEE S&P conference, based on their paper investigating software signing practices across four major public package registries: PyPI, Maven Central, DockerHub, and Hugging Face. The talk, titled "Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors," addresses a critical aspect of software supply chain security by providing an up-to-date, multi-registry empirical analysis of signature adoption and efficacy.

The research provides a much-needed comprehensive perspective, bridging gaps left by prior limited-scale studies that often focused on a single registry. Schorlemmer's presentation illuminates not only the current state of signing – both in terms of how many artifacts are signed (quantity) and how good those signatures are (quality) – but also explores the underlying factors that influence maintainers' decisions to adopt and correctly implement signing. This includes an examination of the impact of registry policies, dedicated tooling, cybersecurity events, and startup costs.

The significance of this work cannot be overstated in the current threat landscape, where software supply chain attacks are increasingly sophisticated and common. Understanding the actual state of provenance and the effectiveness of cryptographic signing in widely used package registries is crucial for developing robust defense strategies. The findings highlight areas of concern, identify effective levers for improvement, and offer actionable insights for registry operators, tool developers, and maintainers striving to secure the software ecosystem.

Background

▶ Watch: Introduction and paper contributions overview (0:00)

The concept of provenance is fundamental to software supply chain security. It refers to the origin and history of an artifact within the supply chain, answering questions like "Where did this come from?" and "Who created it?". In a complex modern software ecosystem, where applications often depend on hundreds or thousands of third-party packages, establishing clear provenance is vital to trust and security. Without it, verifying the integrity and authenticity of software components becomes challenging, leaving users vulnerable to malicious injections or tampering.

A primary technique for establishing provenance is signing, a cryptographic method that provides a verifiable proof of origin and integrity for a software artifact. The process typically involves a sender (Alice) creating a digital signature for a message or artifact using their private key. This signature, along with the artifact, is then sent to a receiver (Bob). Bob, in turn, uses Alice's publicly available public key to verify the signature. If the verification is successful, it guarantees two things: first, that the artifact was indeed created by the specific identity associated with the private key (Alice), and second, that the artifact has not been altered since it was signed. It's important to note that signing does not inherently guarantee the goodness or security of the software itself, only its origin and integrity from the point of signing.

The general framework for software signing, as outlined in the paper, involves a maintainer creating software and generating a signature for it using a public-private key pair. Both the software package, its signature, and the public key are then published to a registry. An end-user downloads these components and uses the public key to verify the signature against the package. However, this process is fraught with potential failure points. The authors categorize these failures into two main trends:

1. Non-cryptographic failures (Orange indicators in the original diagram): These occur when the signer or user fails to properly upload, publish, or discover the signature or key. This includes scenarios where a signature exists but is not easily found or associated with the package.
2. Cryptographic errors (Red indicators): These involve fundamental issues with the cryptographic validity of the signature, such as an expired key, a revoked key, or a cryptographic mismatch between the key and the signature itself.

To understand why maintainers choose to sign, or fail to sign effectively, the researchers developed a signing adoption theory influenced by behavioral economics. This theory posits that various incentives influence a maintainer's decision-making process. First, they decide whether to create a signature at all. If yes, the next decision is whether to create it correctly. A failure at either stage results in an unsigned or poorly signed artifact. The paper identifies four key incentives influencing these decisions:

1. Registry policies: Explicit rules or guidelines from package registries that either encourage or discourage signing.
2. Dedicated tooling: Tools that simplify the signing workflow, reducing friction and complexity for maintainers.
3. Cybersecurity events: Significant incidents like major attacks or the publication of new industry/government standards that might prompt a change in security practices.
4. High startup costs: The initial effort and learning curve associated with creating the very first signature, which can be a significant barrier.

For the purpose of the talk, Schorlemmer focused primarily on the impact of registry policies and cybersecurity events, presenting two core hypotheses:

• Hypothesis 1: Registry policies that explicitly encourage or discourage software signing will have a corresponding direct effect on signing quantity.
• Hypothesis 2: Cybersecurity events, such as cyberattacks or the publication of government/industry standards, will increase signing adoption (both quantity and quality). These hypotheses formed the basis for their empirical investigation into the current state and influencing factors of software signing.

Key Findings

▶ Watch: What is software signing? Cryptographic proof (2:22)

The research yielded critical insights into the landscape of software signing across the selected registries, addressing both the current state of adoption and the factors driving it.

RQ1: Current Quantity and Quality of Signatures

The measurements revealed a stark reality regarding signature adoption:

• Low Signing Quantity: With the notable exception of Maven Central, most registries exhibit extremely low signing rates. In 2023, for instance, PyPI, DockerHub, and Hugging Face all had less than 1% of their artifacts signed. This indicates a widespread lack of basic cryptographic provenance for the vast majority of software packages.
• Varying Signing Quality: Among the artifacts that were signed, a significant proportion suffered from quality issues. For example, PyPI and Hugging Face showed a substantial number of signatures with identifiable problems, meaning they did not provide reliable cryptographic guarantees. DockerHub was an exception, demonstrating a higher rate of "good signatures" among its signed artifacts.
• Outliers Explained: The two registries that deviated from the low adoption trend were Maven Central and DockerHub. Maven Central mandates signing for all published packages, a policy that directly correlates with its consistently high signing rates. DockerHub, on the other hand, provides its own dedicated signing tool, Docker Content Trust, which simplifies the signing process and contributes to both higher adoption and better quality signatures within its ecosystem.

RQ2: Change in Signing Over Time

Analyzing trends over time further elucidated the impact of specific factors:

• Maven Central's Consistent High: Maven Central consistently maintained a high signing rate due to its mandatory policy, demonstrating the power of enforcement.
• DockerHub's Policy-Driven Bump: DockerHub experienced a noticeable increase in signing adoption following an update to its platform that enhanced provenance mechanisms. This highlights how platform-level changes and tooling can drive adoption.
• PyPI's Decline: PyPI showed a declining trend in signing, eventually leading to the removal of its signing capabilities in late 2023. This was directly attributed to the platform's decision to deemphasize PGP signing, indicating that a lack of explicit support or active discouragement can lead to abandonment.
• Hugging Face's Volatility: Hugging Face displayed large spikes in signing between 2017 and early 2020, but these were largely artifacts of a very low sampling rate and package count during that period. After this initial phase, signing adoption dropped to near zero, underscoring a lack of sustained, organic adoption.

RQ3: Influence of Incentives on Signature Adoption

The investigation into incentives yielded some counterintuitive, yet crucial, findings:

• Registry Policies Drive Quantity: Hypothesis 1 was strongly supported. Registry policies that explicitly encourage or mandate signing (like Maven Central's) or discourage it (like PyPI's deemphasis on PGP) have a direct and significant impact on the quantity of signatures. However, these policies do not necessarily improve the quality of those signatures unless combined with effective tooling or simplified workflows.
• Cybersecurity Events Have Minimal Impact: Hypothesis 2, which posited that cybersecurity events would increase adoption, was largely disproven. The study found no statistically significant change in signing adoption (neither quantity nor quality) following major cyberattacks (e.g., the DockerHub hack in April 2019, or the SolarWinds supply chain attack in December 2020) or the publication of new industry standards. This suggests that maintainers are often reactive to direct requirements rather than proactive in response to broader security incidents.
• Tooling and Startup Costs: While not the primary focus of the talk, the paper also indicates that dedicated tooling simplifies the workflow and improves signature quality, while high startup costs act as a barrier to initial adoption. "Getting started is the hard part," as Schorlemmer noted.

In summary, the research paints a picture of nascent and often ineffective signing practices across major registries, heavily influenced by registry-level policies and tooling, but surprisingly resilient to the broader impact of cybersecurity incidents.

Technical Deep Dive

▶ Watch: Common failure modes in software signing (4:00)

The technical depth of this research lies in its rigorous methodology for collecting and analyzing signature data across diverse package ecosystems and its systematic approach to evaluating influencing factors. The selection of four distinct registries was strategic, covering different modalities of software distribution:

• PyPI (Python Package Index): Represents traditional programming language packages, widely used in the Python ecosystem.
• Maven Central: The primary repository for Java libraries, also a traditional software ecosystem.
• DockerHub: Dedicated to container images, representing a modern software deployment paradigm.
• Hugging Face: A newer platform focused on machine learning models and datasets, showcasing an emerging software artifact type.

This diverse selection allowed for a broad assessment of signing practices across different technical communities and operational models. The methodology involved:

1. Package Collection: A comprehensive collection of packages from each registry within a specified time range.
2. Filtering: Packages were filtered to include only those within a defined time frame and with at least five versions, ensuring a dataset suitable for longitudinal analysis.
3. Signature Measurement: For each remaining package, the presence and characteristics of signatures were meticulously measured. This involved identifying signature files, public keys, and evaluating their cryptographic validity.
4. Adoption Evaluation: The adoption of signing was then evaluated along two axes: over time (RQ2) and in relation to the identified influencing factors (RQ3).

The "signing adoption theory" provides a critical lens for this analysis. The failure points in the signing process were detailed, distinguishing between human/operational errors and cryptographic deficiencies. The "orange" failure points – such as failure to upload a signature, publish a key, or for a user to discover them – highlight the importance of user experience and registry integration. The "red" failure points – including expired keys, revoked keys, or cryptographic mismatches – underscore the need for robust public key infrastructure (PKI) management and proper cryptographic hygiene. The prevalence of these failures directly impacts the "quality" metric of signing.

The impact of registry policies was demonstrated through several key observations:

• PyPI's PGP De-emphasis: PyPI's decision to deemphasize PGP signing led to a clear decline in signing quantity. This illustrates that if a registry does not actively support or integrate a signing mechanism, maintainers will naturally drift away from using it, even if the capability technically exists. This effectively removes an incentive for signing.
• DockerHub's Platform Update: An update to DockerHub that enhanced provenance capabilities preceded a bump in signing adoption. While not explicitly a "mandate," this platform improvement likely simplified the process or made signing more visible and integrated, thereby reducing the high startup costs and acting as a form of dedicated tooling.
• Maven Central's Mandate: The stark difference in signing rates between Maven Central (high) and all other registries (low) is the most compelling evidence for the impact of mandatory policies. Maven Central's requirement for users to sign their packages directly translates into near-universal adoption, reinforcing the idea that strong policy enforcement is the most effective lever for increasing signing quantity.

Conversely, the lack of impact from cybersecurity events was a significant finding. For example:

• DockerHub Hack (April 2019): Despite a notable security incident affecting a major registry, the study found that the increase in DockerHub signing occurred much later (about 9 months after the hack), suggesting no direct or immediate causal link.
• SolarWinds Attack (December 2020): This massive, high-profile software supply chain attack, which reverberated across the industry, did not lead to a statistically significant increase in signing adoption on PyPI, DockerHub, or Hugging Face. While Maven Central continued its upward trend, this was attributed to its existing mandatory policy, not a reaction to SolarWinds. This strongly suggests that maintainers, as a collective, do not proactively increase their signing efforts in response to external threats or industry-wide security events. Their behavior is more influenced by direct requirements or simplified tools.

The paper's investigation into dedicated tooling also offers a crucial technical insight: while policies drive quantity, good tooling is essential for quality. The success of Docker Content Trust on DockerHub, for instance, implies that when a user-friendly and well-integrated tool is provided, it can significantly improve the correctness and reliability of signatures, mitigating the "cryptographic error" failure modes. This highlights the importance of abstracting away the complexities of key management and cryptographic operations for the average maintainer.

Demo / Proof of Concept

▶ Watch: Four incentives influencing maintainer signing decisions (6:00)

This research focused on an empirical study and analysis of existing data rather than the development or demonstration of a new tool or exploit. Therefore, the talk did not include a live demo or a proof of concept of a specific vulnerability or defensive mechanism. The presentation was dedicated to sharing the findings from the comprehensive data analysis and the conclusions drawn regarding software signing practices.

Defensive Implications

▶ Watch: Research questions and hypotheses on signing adoption (7:00)

The findings from this research carry profound defensive implications for the entire software supply chain. The current state of software signing, characterized by alarmingly low adoption rates and significant quality issues across most major registries, indicates a widespread vulnerability in the foundational trust mechanisms of our digital infrastructure.

First and foremost, registry operators emerge as the most powerful actors in influencing signing adoption. The stark contrast between Maven Central's high signing rates (due to mandatory policies) and the low rates of other registries demonstrates that explicit policies are the strongest lever for increasing signing quantity. To improve the security posture of their ecosystems, registry operators should seriously consider implementing and enforcing signing requirements. This could range from strong encouragement to outright mandatory signing, depending on the risk profile and community readiness.

However, increasing quantity alone is insufficient; quality is equally critical. The research indicates that while policies drive adoption, dedicated tooling significantly impacts the correctness and reliability of signatures. Registry operators and tool developers should prioritize creating and integrating user-friendly signing tools that abstract away the complexities of key management and cryptographic processes. Tools like Sigstore, which aims to simplify code signing by removing the burden of key management, represent a promising direction to reduce high startup costs and cryptographic failure modes. By making it easy to sign correctly, these tools can improve the overall quality of signatures.

A key takeaway for defenders is the observed disconnect between significant cybersecurity events and maintainer behavior. The fact that major attacks like SolarWinds did not statistically increase signing adoption suggests that relying on fear or awareness campaigns alone is insufficient. Maintainers, often resource-constrained, tend to be reactive rather than proactive. This implies that security improvements related to signing must be integrated into their existing workflows as requirements or simplified tools, rather than being left to individual discretion or post-incident remediation.

For software consumers, the findings underscore the need for caution. The low signing rates mean that the vast majority of packages lack verifiable provenance. Consumers should be aware of this risk and consider implementing additional supply chain security measures, such as vulnerability scanning, dependency analysis, and careful vendor selection, especially for critical components. They cannot solely rely on cryptographic signatures for trust in most ecosystems today.

Ultimately, securing the software supply chain through robust signing will require a multi-pronged approach:

• Policy Enforcement: Registries must implement and enforce mandatory or strongly encouraged signing policies.
• Tooling Simplification: Develop and integrate dedicated, user-friendly signing tools that reduce complexity and improve signature quality.
• Ecosystem Education: While events don't drive adoption, targeted education on how to sign correctly with new tools, combined with policy changes, can help.
• Shift Left in Security: Integrating signing earlier into the development and release pipelines, potentially through CI/CD systems, can make it a default practice.

Without these concerted efforts, the promise of cryptographic provenance for software supply chain security will remain largely unfulfilled.

Key Takeaways

• Widespread Lack of Signing: Software signing practices vary drastically across major package registries, with most platforms (e.g., PyPI, Hugging Face) showing alarmingly low adoption rates (less than 1% in 2023) and significant quality issues among existing signatures.
• Registry Policies are King: Explicit registry policies, particularly mandatory signing requirements like those in Maven Central, are the most effective non-technical lever for significantly increasing the quantity of signed software artifacts.
• Cybersecurity Events Have Limited Impact: Major cyberattacks (e.g., DockerHub hack, SolarWinds) and the publication of new security standards have little to no statistically significant influence on maintainers' decisions to adopt or improve software signing practices. Maintainers are generally reactive to direct requirements, not proactive to external threats.
• Tooling Drives Quality and Reduces Barriers: Dedicated, user-friendly signing tools (like Docker Content Trust) are crucial for improving the quality of signatures and reducing the "high startup costs" associated with initial signing, which often acts as a barrier to adoption.
• Sigstore Offers Promise: Emerging tools like Sigstore are critical for improving the future state of software signing by simplifying the complex key management overhead, which is a major source of cryptographic failure modes.
• Urgent Need for Action: The current state of software signing is insufficient for providing robust provenance and securing the software supply chain, necessitating a concerted effort from registry operators, tool developers, and the community to mandate, simplify, and integrate signing practices.

About the Speaker(s)

The primary presenter for this talk was Taylor R. Schorlemmer, who delivered the findings as part of their Master's thesis work. This research was a collaborative effort with co-authors from Purdue University, including Dr. James C. Davis (thesis advisor), Kelechi G. Kalu, Luke Chigges, Kyung Myung Ko, Eman Abdul-Muhd Abu Ishgair, and Saurabh Bagchi. The project received support from notable organizations such as Cisco, Google, and the National Science Foundation, with additional assistance from the Purdue Military Research Institute. The collective expertise of this team, spanning academic research and security, underpins the comprehensive and analytical nature of the study presented.