Where The Wild Things Are: The State Of Open Source Supply Chain Risk Management In Three Stories

Where The Wild Things Are: The State Of Open Source Supply Chain Risk Management In Three Stories

Speakers: Maui, Speaker, (Company name not explicitly stated in talk)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=ZsnOt5o_2HM

Overview

In this insightful talk at VulnCon, "Where The Wild Things Are: The State Of Open Source Supply Chain Risk Management In Three Stories," speaker Maui delves into the critical challenges facing modern software supply chains, particularly the pervasive and often hidden risks associated with open-source dependencies. The presentation highlights that the vast majority of software applications are not proprietary but are built upon a complex ecosystem of open-source components, inherently inheriting their vulnerabilities and risks. Maui argues that current approaches to supply chain security are predominantly reactive, leaving organizations vulnerable to long-latent bugs and overwhelmed by a deluge of non-actionable alerts.

Maui structures the talk around "three wild things" that represent the core problems in open-source supply chain risk management: the reactive nature of existing Software Composition Analysis (SCA) tools, the lack of actionable advice for identified vulnerabilities, and the overwhelming alert fatigue caused by the sheer volume of CVEs and their often-unknown reachability. The speaker advocates for a paradigm shift towards proactive security measures, emphasizing the need for advanced static analysis, comprehensive risk data, and a more humane, collaborative approach to engaging with open-source maintainers. The talk is crucial for anyone involved in software development, security operations, or risk management who relies on open-source software and seeks to understand and mitigate its inherent complexities and dangers.

Background

▶ Watch: Introduction to open source supply chain risk (0:00)

The foundation of modern software development is irrevocably tied to open-source components. As Maui highlights, a staggering 74% of codebases harbor high-risk open-source vulnerabilities, a statistic that underscores the ubiquitous and often unmanaged risk inherent in today's software ecosystem. This reliance on open source means that organizations are not just consuming code, but transitively inheriting the security posture, or lack thereof, of countless upstream projects.

A significant hurdle in managing this risk is the fundamentally reactive nature of traditional Software Composition Analysis (SCA) tools. These tools typically spring into action only after a bug has been discovered, reported, and assigned a Common Vulnerabilities and Exposures (CVE) identifier. By this point, the vulnerability may have existed in the codebase for a considerable duration, presenting a substantial window for exploitation. Maui illustrates this critical latency with several historical examples:

• Heartbleed, a critical vulnerability in OpenSSL, was present in the code for approximately two years before its discovery in 2014.
• Log4Shell, the infamous vulnerability in Apache Log4j, was latent for about four years—a feature introduced in 2013 only reported as a vulnerability in 2021 after active exploitation was detected by Alibaba researchers.
• Windows PrintNightmare, a bug in the Windows print spooler, resided in the source code for an astonishing 24 years, from 1996 until its eventual fix in 2020.

These examples vividly demonstrate that the bugs organizations are currently scrambling to fix have often been "happening" for years, potentially remaining unexploited but posing a persistent threat. The speaker, drawing on a background in academia and industry with experience in static analysis and bug detection, including leading the team that released the first commercial vulnerability detection tool for JavaScript and Android at Coverity, emphasizes the urgent need to shift from this reactive posture to a proactive one. The challenge, however, is immense: if finding bugs in proprietary code is difficult, scaling this effort to thousands of open-source dependencies underneath an application presents a formidable task. This forms the impetus for the innovative approaches discussed in the talk.

Key Findings

▶ Watch: Identifying the reactive nature of SCA tools (4:40)

The talk presents several key findings and contributions aimed at addressing the reactive nature of open-source supply chain security and the pervasive alert fatigue. Central to these findings is the demonstration of proactive bug finding at scale and a more collaborative approach to vulnerability disclosure.

Through funding from Alpha Omega, Maui's team embarked on an ambitious project to proactively scan the top 2,000 Python projects on PyPI and their extensive network of dependencies. This effort, encompassing approximately 14,000 distinct dependencies, yielded significant results:

• Discovery of about 350 bugs in total.
• Identification of 150 security-specific bugs.
• Pinpointing 55 high-severity security bugs.
• A commendable 65% fix rate for reported vulnerabilities, a testament to their engagement strategy.

This success stands in stark contrast to previous large-scale bug reporting efforts, such as the "bulk pull request" approach. Maui references work by Jonathan Lechu (2019-2021) and, more notably, Trellis researchers in 2022, who attempted to generate 61,000 patches for a CVE dating back to 2007. While impactful in scale, such methods often generate significant "anger from the maintainers" due to a fundamental disconnect between security researchers and open-source project stewards. Maintainers frequently feel overwhelmed by reports lacking context, empathy, or follow-through, perceiving them as "drive-by PRs."

Maui's team addressed this by adopting a more humane and collaborative reporting process. Instead of automated, anonymous submissions, reports are made from an "actual human handle" that commits to back-and-forth communication. This includes helping maintainers validate bugs by writing exploit code and even assisting in creating patches on their behalf. This hands-on, empathetic approach is a critical factor behind the high fix rate achieved.

Further demonstrating the efficacy of their proactive methodology, Maui introduced Project Clean Beach, a data feed service designed to provide downstream consumers with actionable intelligence about their specific open-source dependencies. This service offers insights into known vulnerabilities, risks from human factors (e.g., project activity), and crucially, previously undetected bugs. A case study with Apache Airflow, which has 719 dependencies, revealed 16 new bugs, including 4 high-severity issues, along with the identification of unused dependencies, aiding in cleanup efforts.

Another significant finding emerged from a study on Jenkins (version 2.476), involving 172 artifacts. This analysis uncovered 18 new high-severity bugs across 12 risky artifacts, representing two times more risky artifacts than identified by traditional SCA tools. This highlights the limitations of current tools in finding latent, unknown vulnerabilities. The team also developed the "Six Fs" framework to provide concrete, actionable advice for mitigating these findings, moving beyond mere bug reports to practical solutions.

Finally, the talk touches upon the critical problem of alert fatigue and the potential of Vulnerability Exploitability eXchange (VEX) documents. Maui cites research (from Simg) indicating that 97% of SCA reports can be noise, meaning the reported CVEs do not actually impact the specific application due to lack of reachability. The speaker's team is developing capabilities to automatically generate VEX documents, aiming to address this scalability challenge that typically renders manual VEX generation (e.g., for Apache Solar, requiring an estimated 0.9 person-years for 900 VEX documents) impractical for most organizations. This capability promises to dramatically reduce developer toil and improve incident response by providing timely, accurate reachability analysis.

Technical Deep Dive

▶ Watch: Famous vulnerabilities (Heartbleed, Log4Shell) latent for years (5:40)

The technical core of Maui's presentation revolves around overcoming the limitations of traditional static analysis tools and scaling proactive security measures. The speaker introduces ICR (Intelligent Code Repair), their proprietary static analysis tool, as the foundation for their bug-finding capabilities.

Intelligent Code Repair (ICR): The Engine for Proactive Discovery

A primary challenge with static analysis tools is their propensity for false positives, which render them impractical for large-scale, automated scanning of thousands of open-source dependencies. Maui vividly illustrates this problem with a comparative study on Kubernetes. When pitted against five other static analysis tools (CodeQL, Semgrep, Snyk, SonarCloud), ICR demonstrated superior accuracy:

• None of the other tools were able to find a seeded bug intentionally placed in a specific Kubernetes version for testing.
• Conversely, some tools generated an astonishing 16,244 reports, all of which were later identified as false positives.

ICR's ability to minimize false positives is critical, as it allows the team to "look at these at scale and also had time to help with the PRs to help with creating the PC exploit codes and following the norms." This accuracy is what enables their scalable bug-finding and collaborative reporting strategy, making the output actionable rather than overwhelming.

Project Clean Beach: A Data Feed for Comprehensive Risk Management

Building on ICR's capabilities, Project Clean Beach is presented as a crucial data feed service. Unlike typical SCA tools that focus solely on known CVEs, Project Clean Beach generates a holistic risk management signal by:

1. Ingesting a Software Bill of Materials (SBOM).
2. Providing information on known bugs in the supply chain.
3. Assessing risk from human factors (e.g., project activity, maintainer responsiveness).
4. Uniquely, identifying risk from previously undetected bugs found by ICR.

This proactive intelligence allows downstream consumers to "get ahead of time intel," react better to newly discovered bugs, and potentially prevent vulnerabilities from being shipped in released applications, saving millions of dollars. The service also considers code provenance, offering insights into where code is consumed from.

A specific measure of their tool's thoroughness is the tracking of missed high-severity bugs (false negatives). By focusing on approximately 60 Common Weakness Enumerations (CWEs) and re-scanning projects, the team built a live tracker. Maui proudly reports that in all their extensive scanning efforts across thousands of projects, they have missed only one high-severity bug in 2024. This particular instance, a cross-site scripting (XSS) in a Python project, was missed because it involved a custom request object that their taint analysis initially failed to identify as an external input source. Once provided with additional metadata about this custom request, ICR was able to detect it, highlighting the continuous refinement of their tooling.

The Six Fs: Actionable Mitigation Framework

To translate bug findings into concrete actions, Maui introduces the Six Fs framework, a categorization of mitigation strategies:

1. Fix: Upgrade to a newer version where the bug is already resolved. (Moderate effort)
2. Flip: Replace a problematic package with an equivalent one. (Higher effort)
3. Forge: Collaborate with open-source maintainers to get a fix developed and released upstream. (High effort, community contribution)
4. Fork: Create a private fork to apply a local fix, hoping for an upstream resolution later. (Moderate effort)
5. Forget: Accept the risk or ignore the issue, often due to no available options or low impact. (Low/no effort)
6. Forego: Abandon the package entirely and build the functionality in-house. (High effort, significant investment)

This framework provides a structured approach for organizations to tackle the 18 new high-severity bugs and 29 known high-severity bugs found in the Jenkins study. For instance, the study revealed that for 60% of risky artifacts, a fix was already available in a future version (categorized as "Fix"), while others required "Forge" (working with maintainers) or "Flip/Fork" for archived packages. For each "Fix" scenario, the team provides a specific patch or upgrade path.

VEX Document Generation: Tackling Alert Fatigue

The third "wild thing" addresses the overwhelming alert fatigue from CVEs. Maui explains that most reported CVEs may not actually be "reachable" or exploitable within a specific application's context. The solution lies in Vulnerability Exploitability eXchange (VEX) documents, which clarify whether a product is affected by a known vulnerability.

Maui demonstrates the value of VEX with a real-world example: Apache Kafka's dependency chain (Kafka -> Log4j -> Jackson Data Bind -> SnakeYAML). SnakeYAML had eight CVEs. A manual reachability analysis initially suggested seven reached Jackson Data Bind and zero reached Log4j or Kafka. However, their automated tooling provided a more accurate picture, concluding that zero CVEs from SnakeYAML actually reached Kafka. This stark difference highlights the power of automation over manual analysis, which "does not scale" and "is already problematic because it does not help you in your large projects."

The team's goal is to automate VEX document generation within 72 hours of a CVE's public release. This involves:

• Identifying the root cause function for a CVE within the vulnerable package.
• Generating call graphs for the entire supply chain using ICR.
• Performing reachability analysis across each hop (e.g., A to B, B to C, C to D).
• Providing VEX documents for each step.

The proposed delivery mechanism integrates with GitHub Actions, triggering API requests for VEX documents upon Dependabot alerts, followed by private email disclosures to maintainers (after an embargo period to prevent side-channel attacks), and eventually public PRs for the VEX documents. This comprehensive approach aims to eliminate 97% of unnecessary toil by providing precise, timely, and actionable information on true exploitability.

Demo / Proof of Concept

▶ Watch: Proposing proactive vulnerability discovery via Alpha Omega (7:20)

While the presentation did not feature a live, interactive demonstration in the traditional sense, Maui extensively described several real-world case studies and project outcomes that serve as robust proof-of-concept for the methodologies and tools discussed. These "demonstrations" illustrate the practical application and effectiveness of their approach to open-source supply chain security.

1. Proactive Bug Finding in PyPI: The project funded by Alpha Omega, which scanned the top 2,000 Python projects and their 14,000 dependencies, served as a large-scale demonstration of the ICR (Intelligent Code Repair) tool's ability to discover previously unknown bugs. The identification of 350 bugs, including 55 high-severity security issues, with a 65% fix rate, concretely proved the viability of proactive, accurate static analysis at an unprecedented scale for open-source ecosystems. The contrast with the "bulk pull request" approach highlighted the success of their "humane" and collaborative disclosure process.

1. Project Clean Beach with Apache Airflow: This case study showcased the practical implementation of the Project Clean Beach data feed. By working directly with Apache Airflow maintainers and analyzing its 719 dependencies, the team demonstrated the capability to identify 16 new bugs (4 high severity) and numerous risky artifacts. This "demonstration" also revealed valuable byproducts, such as identifying and removing unused dependencies, further validating the comprehensive risk intelligence provided by the service.

1. Actionable Advice for Jenkins: The analysis of Jenkins (version 2.476) and its 172 artifacts served as a proof-of-concept for both the discovery of unknown bugs and the application of the Six Fs mitigation framework. The finding of 18 new high-severity bugs and two times more risky artifacts than other SCA tools underscored the depth of their analysis. Furthermore, categorizing these findings under "Fix," "Forge," and "Fork" provided a tangible demonstration of how actionable advice is generated and presented to downstream consumers, moving beyond mere alerts to concrete steps.

1. VEX Reachability Analysis with Apache Kafka: The example involving Apache Kafka, Log4j, Jackson Data Bind, and SnakeYAML effectively demonstrated the power and accuracy of their automated VEX document generation and reachability analysis. By contrasting manual analysis (which suggested 7 CVEs reached Jackson Data Bind) with their tooling (which found 0 CVEs from SnakeYAML reached Kafka), Maui provided compelling evidence that automated, precise call graph analysis can dramatically reduce false positives and alert fatigue, offering a reliable answer to the critical "does this CVE impact me?" question.

These described "demos" collectively illustrate that the speaker's methodologies and tools are not theoretical but have been successfully applied to real-world, complex open-source projects, yielding concrete, impactful security improvements and a path forward for better supply chain risk management.

Defensive Implications

▶ Watch: Initial scorecard results from 2,000 Python projects (8:20)

The insights shared by Maui offer several critical defensive implications for organizations grappling with open-source supply chain risk:

1. Shift to Proactive Security: Defenders must move beyond a purely reactive stance, which relies on CVEs being issued. Investing in proactive bug finding capabilities, particularly through advanced static analysis tools like ICR that boast low false positive rates, is paramount. This enables the discovery of latent vulnerabilities before they are exploited, significantly narrowing the window of exposure.

1. Leverage Comprehensive Risk Data Feeds: Organizations should seek out and integrate data feeds such as Project Clean Beach into their security decision-making systems. These feeds provide a more holistic view of risk, encompassing not only known CVEs but also previously undetected bugs and insights into "human factors" like project activity and maintainer responsiveness. This allows for more informed decisions when ingesting new open-source components and managing existing ones.

1. Prioritize Vulnerabilities with Reachability Analysis (VEX): The overwhelming volume of SCA alerts leads to alert fatigue. Defenders should prioritize vulnerabilities based on their actual reachability within their specific application context. Adopting or supporting tools that can generate VEX documents rapidly (e.g., within 72 hours of a CVE's release) will drastically reduce noise, allowing security teams to focus resources on truly impactful threats. This capability can eliminate up to 97% of irrelevant alerts, as demonstrated by research.

1. Implement Structured Mitigation Strategies: Instead of ad-hoc responses, organizations should adopt a structured framework for addressing vulnerabilities. The Six Fs (Fix, Flip, Forge, Fork, Forget, Forego) provide a clear decision tree for remediation, guiding teams through upgrading dependencies, replacing problematic packages, contributing upstream, or making strategic decisions about abandoning components. This systematizes the response process, making it more efficient and effective.

1. Foster Upstream Contribution and Compassion: Recognize that open-source projects are maintained by dedicated individuals, often without significant resources. As consumers of open source, organizations have a responsibility to contribute back. Engaging in "forging" relationships with maintainers, providing well-contextualized bug reports, and even contributing code fixes or exploit proofs can significantly improve the security posture of shared dependencies. This empathetic and collaborative approach benefits the entire ecosystem.

1. Monitor Longitudinal Security Effort: When evaluating new or existing open-source dependencies, look beyond current CVE counts. Utilize data that tracks the historical trend of CVE resolution and security team activity (e.g., downward slopes in outstanding CVEs). This provides assurance about a project's commitment to security and its ability to handle future vulnerabilities effectively.

By adopting these defensive strategies, organizations can move towards a more mature, proactive, and sustainable approach to managing the inherent risks in their open-source supply chains.

Key Takeaways

• Proactive Bug Finding is Essential: Relying solely on reactive SCA tools after CVEs are issued leaves organizations vulnerable to bugs that have been latent for years. Proactive, scalable static analysis with low false positives is crucial for discovering unknown vulnerabilities before exploitation.
• Actionable Advice is Critical for Mitigation: Raw vulnerability reports are insufficient. Organizations need clear, actionable guidance on how to fix issues. Frameworks like the "Six Fs" (Fix, Flip, Forge, Fork, Forget, Forego) provide structured approaches to efficiently address identified problems.
• VEX Documents Reduce Alert Fatigue: The vast majority of CVE alerts may not actually impact a specific application. Automated generation of VEX documents, coupled with precise reachability analysis, can filter out up to 97% of irrelevant noise, allowing security teams to focus on genuine threats.
• Accuracy Enables Scale: Traditional static analysis tools often generate too many false positives, making large-scale scanning of open-source dependencies untenable. Tools like ICR, with their high accuracy, make it possible to proactively find bugs across thousands of projects without overwhelming maintainers or security teams.
• Humanity and Collaboration Drive Open Source Security: Effective open-source security requires empathy and collaboration with maintainers. Engaging in well-communicated, contextualized bug reporting, assisting with exploit proofs, and contributing to upstream projects fosters trust and leads to higher fix rates.
• Data Over Dashboards: Instead of prescriptive dashboards, organizations need comprehensive, raw data feeds that integrate into their existing security decision-making systems. This empowers them to tailor risk management strategies based on their unique context and requirements, including insights into known, unknown, and human-factor risks.

About the Speaker(s)

The speaker, Maui, brings a wealth of experience from both academia and industry to the field of software supply chain security. Currently, Maui is involved in a startup focused on supply chain security and static analysis for bug detection. Prior to this venture, Maui held a significant role at Coverity, a leading bug detection company, where he led a team responsible for releasing the first commercial vulnerability detection tool specifically designed for JavaScript and Android. His academic background also centered on the research and development of automated tools for detecting bugs in software, providing a strong foundation for his current work in proactive vulnerability identification and mitigation within open-source ecosystems.