Context Matters: Qualitative Insights into Developers’ Approaches and Challenges with Software...

Context Matters: Qualitative Insights into Developers’ Approaches and Challenges with Software Composition Analysis Tools

Speakers: Elizabeth, PhD Student, NC State

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=g-SYh9v3W5Y

Overview

In this insightful talk from VulnCon, Elizabeth, a PhD student from NC State's Whisper Lab, presented a qualitative study examining the real-world experiences of developers interacting with Software Composition Analysis (SCA) tools. The presentation highlighted critical challenges and opportunities for improvement in how these tools are integrated into development pipelines and how their outputs are interpreted and acted upon. The core message underscored that "context matters" significantly, yet is often lacking in current SCA solutions, leading to developer frustration and inefficiencies.

The talk provides a deep dive into why SCA tools are essential, using the notorious Log4Shell vulnerability as a prime motivator. SCA tools are designed to identify risks and vulnerabilities within external libraries and transitive dependencies—code not written by the development team itself. However, as the study reveals, the journey from identifying a potential vulnerability to effectively mitigating it is fraught with complexities, largely due to the absence of crucial contextual information. This research offers a vital perspective for both SCA vendors aiming to enhance their products and development teams striving to better integrate these security measures.

Background

▶ Watch: Introduction to SCA and supply chain security (0:00)

The proliferation of open-source components and third-party libraries in modern software development has made Software Composition Analysis (SCA) tools indispensable. The Log4Shell incident in 2021 starkly illustrated the need for robust SCA, as organizations scrambled to identify and remediate instances of the vulnerable Log4j library, often buried deep within complex dependency trees. An ideal SCA tool, as envisioned by the speaker, should deploy smoothly, correctly identify components, alert only on vulnerabilities that truly matter, and provide actionable fix suggestions.

However, the reality often falls short of this ideal. Elizabeth demonstrated common pain points by showcasing a typical SCA scan result: an overwhelming number of alerts, often hundreds of critical or high-severity findings, making prioritization and remediation a daunting task. Generic CVE information frequently lacks the specific context needed to determine actual impact or exploitability within a particular application. Developers are left to navigate intricate dependency paths, often involving transitive libraries introduced by multiple direct dependencies, and grapple with a multitude of ambiguous fix suggestions. This pervasive lack of context necessitates significant manual effort, leading to frustration and, in many cases, an inability to effectively address identified vulnerabilities. It was this gap between the ideal and the reality that motivated the qualitative user study, aiming to understand the underlying challenges faced by industry professionals.

Key Findings

▶ Watch: Four characteristics of an ideal SCA tool (4:20)

The central finding of the research, derived from interviews with 20 industry professionals, is that context really matters across the entire SCA tool process, and its absence is a significant impediment. This overarching theme manifests in three key aspects:

1. Lack of Context in Vulnerability Alerts: SCA tools typically provide generic CVE information, leaving developers to manually determine if their specific application is impacted or exploitable. This requires substantial extra effort and expertise to assess true risk.
2. Tradeoffs in Integration Methods: The method of integrating SCA tools into the development lifecycle (e.g., automated CI pipeline scans vs. manual periodic checks) presents different tradeoffs. Automated scans can introduce significant "noise," leading to frequent build failures and pauses in the pipeline, which developers find disruptive. This sometimes pushes teams back to less efficient manual scanning.
3. Communication Overhead Across Teams and Supply Chains: In larger organizations or across complex supply chains, communicating SCA results between different teams (security, engineering, legal, suppliers) becomes challenging. Disparate workflows, toolsets, and contextual understanding among parties lead to communication overhead and can hinder effective remediation.

Beyond these core findings, the study also uncovered insights into how users interact with SCA tools. Participants primarily adopted SCA for identifying software vulnerabilities and ensuring compliance/licensing. Interestingly, some also leveraged SCA for less common use cases, such as due diligence during mergers and acquisitions (M&A) to understand a target company's software stack, and for export controls to ensure compliance with strict regulations on software components. Ease of deployment, for instance, integrating with existing platforms like GitHub via tools like Dependabot, emerged as a major factor in tool selection. Accuracy of results and flexible scan requirements were also critical, with participants often switching tools if they encountered high rates of false positives or overly rigid configurations.

Technical Deep Dive

▶ Watch: Real-world SCA scan: Overwhelming alerts (6:15)

The qualitative study elucidated several technical intricacies and challenges inherent in developers' interactions with SCA tools, spanning input methods, deployment, and remediation.

Participants reported three primary input methods for SCA tools:

• Source Code: The most common method, where the entire application source code is provided to the SCA tool. This raises concerns for some organizations regarding sensitive code leakage, especially when external cloud-based SCA services are used.
• Metadata: To mitigate source code leakage concerns, some tools accept only metadata, such as package.json or manifest files. This provides a less detailed but often sufficient view of direct dependencies without exposing proprietary code.
• Binaries: When organizations acquire pre-compiled binaries from suppliers without access to source code, binary SCA scans are employed. However, participants noted that binary SCA tools often struggle with correctly identifying library versions due to reliance on fingerprinting or hashing. Minor code changes (e.g., a single variable name) can alter fingerprints, confusing the tool, especially with copy-pasted code segments.

Deployment challenges were significant, particularly in diverse technical environments:

• Legacy Languages and Unsupported Ecosystems: Many SCA tools lack comprehensive support for less common package managers or older programming languages. This forces teams to find workarounds or engage directly with vendors for custom solutions.
• Multi-Ecosystem Applications: A common issue arises when applications incorporate multiple language ecosystems (e.g., Java backend, Node.js frontend). A single SCA tool rarely covers all ecosystems effectively. Integrating multiple tools means writing custom wrappers or scripts to harmonize their outputs, leading to substantial maintenance overhead when any tool updates.
• CI Pipeline Integration: While automated SCA scans in CI pipelines are efficient, they frequently introduce "noise." Overly strict configurations can lead to build failures for every detected alert, causing disruptive pauses. Participants cited instances where the frequency of build breaks forced them to abandon automated CI integration in favor of manual, less effective scans. The challenge lies in intelligently configuring alert thresholds and communicating expected behavior to developers.

Acting on SCA results presented further technical hurdles:

• Prioritization: With a deluge of alerts, prioritization is crucial. While metrics like CVSS scores are used, developers found it challenging to accurately assess the real "time effort needed" and "impact" without additional context.
• Determining Impact and Exploitability: This was a major pain point. Developers need to know if a vulnerable library is actually reachable within their application's execution flow (e.g., through static function reachability analysis). Furthermore, the application's network posture (e.g., sitting behind a firewall) and the presence of untrusted data flow are critical for assessing exploitability. Without this context, alerts for vulnerabilities in firewalled applications or those not exposed to untrusted input are often irrelevant. The speaker emphasized that lawyers often require clear verification of impact, which current SCA outputs rarely provide.
• Fixing Vulnerabilities: The most common fix, updating a vulnerable library, can introduce breaking changes, especially across major versions with different APIs. This necessitates significant refactoring or even custom fixes, such as forking a library to patch out the vulnerable component, which demands considerable engineering effort.
• Reasons for Not Fixing: Technical and business realities sometimes prevent immediate remediation. This includes a lack of actual impact, a strategic decision to weigh security risk against business risk (e.g., delaying a fix to meet a critical release deadline), or the simple unavailability of a non-vulnerable alternative from a sole-source supplier. In such cases, mitigation strategies like isolating the component or removing network access become crucial.

A key suggestion for improvement involved combining SCA with SAST (Static Application Security Testing) tools. This integration could facilitate call graph analysis and data flow analysis, providing the necessary context to determine if a vulnerable function is actually invoked and if untrusted data can reach it, thereby significantly improving alert relevance and prioritization.

Demo / Proof of Concept

▶ Watch: Confusing fix suggestions for transitive dependencies (7:40)

While this talk did not feature a live demonstration or proof-of-concept of a novel exploit, the speaker effectively illustrated the practical challenges developers face by walking through their own experience running an SCA scan on a forked GitHub repository. Elizabeth presented screenshots of the overwhelming number of alerts, the generic nature of CVE information, the complexity of dependency paths leading to vulnerable transitive libraries, and the ambiguity of multiple fix suggestions. This practical walkthrough served as a compelling "proof of concept" for the problem statement, vividly demonstrating the lack of context and the resulting confusion and effort required from a developer trying to interpret and act on SCA results.

Defensive Implications

▶ Watch: User study approach to understand developer challenges (9:00)

The findings of this study provide crucial insights for organizations seeking to improve their software supply chain security posture, particularly concerning the effective use of SCA tools.

For SCA users (developers, security teams, and organizations):

• Strategic Tool Selection: Understand that no single SCA tool is perfect. Organizations should carefully evaluate tools based on their specific application stack, supported languages, package managers, and integration capabilities (e.g., ease of integration with GitHub, support for multiple ecosystems). Prioritize tools that align with your existing development workflows and can minimize the need for custom wrappers or extensive maintenance.
• Intelligent CI/CD Integration: When integrating SCA into CI pipelines, configure alert thresholds judiciously. Instead of failing builds on every low or medium alert, prioritize critical and high-severity issues with confirmed impact. Communicate these thresholds and expected behaviors clearly to development teams to manage expectations and prevent pipeline disruptions from becoming a source of frustration.
• Focus on Contextual Prioritization: Move beyond raw CVSS scores. Implement a prioritization framework that considers actual impact and exploitability. This involves assessing static function reachability, understanding the application's network exposure (e.g., behind a firewall), and analyzing potential untrusted data flow to the vulnerable component. Investing in internal expertise to perform these contextual analyses is paramount.
• Enhance Cross-Team Communication: Recognize that different teams (engineering, security, legal, SRE) have varying contexts and priorities. Establish clear communication channels and processes for discussing SCA results, especially when making decisions about remediation or acceptance of risk. Custom tooling for pre-processing and triaging alerts, as observed in large organizations, can facilitate this by providing a unified, contextualized view for all stakeholders.
• Embrace Mitigation Strategies: Acknowledge that not all vulnerabilities can be immediately fixed by updating. Develop and implement robust mitigation strategies such as isolating vulnerable components, removing network access, or applying compensating controls when direct remediation is not feasible due to technical constraints or business decisions.

For SCA vendors (and the broader security tooling ecosystem):

• Prioritize Contextual Intelligence: The most significant improvement would be to embed more contextual intelligence directly into SCA tools. This includes integrating SAST-like capabilities (e.g., call graph analysis, data flow analysis) to determine if vulnerable functions are reachable and exploitable within the application's specific usage.
• Infrastructure-Aware Analysis: Incorporate infrastructure context, such as firewall configurations and application interaction patterns, to filter out non-impactful alerts.
• Actionable Fix Suggestions: Provide more intelligent and actionable fix suggestions, potentially considering dependency conflicts and API compatibility, to guide developers towards solutions that minimize breaking changes.
• User Feedback Loops: Implement features that allow users to provide feedback on false positives or irrelevant alerts. This creates a feedback loop for continuous improvement, enabling tools to learn from user input and reduce "noise" over time.

By adopting these defensive implications, organizations can transform SCA from a source of overwhelming alerts into a more effective, context-aware, and integrated component of their secure development lifecycle.

Key Takeaways

• Context is Paramount: The primary finding is that a lack of contextual information in SCA alerts significantly hinders developers' ability to assess true risk and effectively prioritize vulnerabilities.
• Overwhelming Alerts Require Intelligent Prioritization: Developers face an unmanageable volume of generic alerts. Prioritization must move beyond CVSS scores to include actual impact, exploitability, network reachability, and untrusted data flow analysis.
• Deployment Challenges are Diverse: SCA tool deployment is complicated by legacy languages, multi-ecosystem applications, and the "noise" generated in CI pipelines, often forcing teams to choose between automation and development velocity.
• Remediation is Complex and Not Always a Simple Update: Fixing vulnerabilities often involves more than just updating libraries, potentially requiring custom patches, dealing with breaking changes, or making strategic decisions to mitigate rather than fix due to business or technical constraints.
• Communication Across Teams is Crucial: Effective software supply chain security requires seamless communication and shared context between security, engineering, and other stakeholders, which is challenging in large organizations with disparate tools and workflows.
• Future SCA Tools Need Deeper Contextual Integration: The ideal SCA tool will combine component analysis with SAST-like data flow, call graph analysis, and infrastructure context to provide highly relevant and actionable insights, potentially learning from user feedback.

About the Speaker(s)

Elizabeth is a PhD student at NC State, currently in her third year of the program. She is part of the Whisper Lab within NC State's Secure Computing Institute, where her research broadly focuses on security and privacy, with a significant emphasis on software supply chain security. Her work has spanned various projects, including research on Software Bill of Materials (SBOMs), vulnerability fixing commits, and vulnerabilities in VS Code extensions. The Whisper Lab actively contributes to a multi-university effort aimed at improving the security posture across the entire software supply chain and engages in outreach programs like cybersecurity camps and industry conferences. Elizabeth is nearing the completion of her PhD and is open to research collaborations and other opportunities.