Determining Exploitability of Vulnerabilities with SBOM and VEX

Determining Exploitability of Vulnerabilities with SBOM and VEX

Speakers: Shina, Security Software Engineer, Splunk; Anushia, Security Software Engineer, Splunk

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=WEbGWfrp1V4

Overview

In an era dominated by open-source software, managing the deluge of associated vulnerabilities has become a paramount challenge for organizations. This talk, presented by Shina and Anushia, security software engineers at Splunk, addresses a critical pain point in modern software development: effectively determining the exploitability of reported vulnerabilities. They detail Splunk's journey in implementing a robust, centralized security practice that leverages Software Bill of Materials (SBOM) and Vulnerability Exploitability eXchange (VEX) to streamline vulnerability management, enhance developer experience, and provide actionable security insights.

The core problem tackled by the speakers is the overwhelming volume of vulnerability findings generated by security tools, a significant portion of which are often not exploitable or actionable in a specific product context. Splunk's internal analysis revealed that over 26% of identified vulnerabilities fell into this category, leading to developer "turmoil" and wasted effort. To combat this, Splunk developed a comprehensive system that integrates asset inventory, centralized scanning, automated SBOM generation, and a novel approach to VEX data collection through existing issue tracking systems.

This presentation offers invaluable insights for any organization grappling with the complexities of open-source security, regulatory compliance (such as the US Executive Order mandating SBOMs for federal contractors), and the need to shift from reactive vulnerability reporting to proactive, context-aware risk management. By sharing their practical implementation strategy, Shina and Anushia demonstrate how a well-architected system can transform a chaotic flood of alerts into a manageable, intelligent, and developer-friendly security workflow.

Background

▶ Watch: Introduction to SBOM, VEX, and open source vulnerability problem (0:40)

The pervasive adoption of open-source components in software development, while accelerating innovation and efficiency, inherently introduces a significant attack surface. Developers frequently opt for open-source libraries to avoid "reinventing the wheel," allowing them to focus on core product features. However, this convenience comes with the inherent risk of inheriting vulnerabilities present in these third-party dependencies.

Historically, organizations have relied on Software Composition Analysis (SCA) tools to detect open-source and third-party components within their products. Early implementations often involved a "shift-left" approach, integrating scanning directly into the CI/CD pipeline. In Splunk's 2022 workflow, developers would push code, CI/CD would trigger scans, results went to product security for review, and feedback was sent back to developers for remediation. While this process was self-service and allowed for customized scan configurations, it suffered from a critical limitation: the lack of a centralized, product-wide view of vulnerability information. Product security teams found it challenging to aggregate data, leading to fragmented insights and operational inefficiencies.

The turning point for Splunk came with the US Executive Order mandating SBOMs for products used by the federal government. This regulatory push served as a catalyst for Splunk to address their existing data gaps and move towards a centralized model. Recognizing the strategic importance of a comprehensive asset inventory, Splunk embarked on building a system that could not only generate SBOMs but also provide a holistic view of their software supply chain.

However, even with improved visibility and automated scanning, a persistent and growing challenge remained: the sheer volume of reported vulnerabilities. As Anushia highlighted, security tools frequently generate a "flood of vulnerabilities," which, if all converted into tickets, leads to a "flood of tickets." Crucially, a significant portion of these findings — Splunk's internal analysis indicated over 26% — were either not exploitable in their specific product context or not actionable (e.g., no available fix, or the vulnerable code path was unreachable). This high rate of false positives and non-actionable alerts caused developer frustration and eroded trust in the security tooling, ultimately hindering efficient remediation efforts. The problem was exacerbated by the continuously increasing number of CVEs released each year, signaling that this challenge was only set to worsen. This context underscored the urgent need for a mechanism to determine the true exploitability of vulnerabilities, leading Splunk to explore Vulnerability Exploitability eXchange (VEX) as a solution to enhance developer experience and focus remediation efforts on genuine risks.

Key Findings

▶ Watch: Shifting to a centralized model and building asset inventory (2:55)

The talk highlights several key findings and contributions from Splunk's journey to improve vulnerability management:

1. Centralized Asset Inventory and Scan Platform as Foundation: Establishing a robust, centralized asset inventory and scan platform is fundamental. This infrastructure enables the seamless generation of SBOMs, provides comprehensive product-level telemetry, and serves as the single source of truth for all software components and their associated scan results. This move from a decentralized, shift-left model to a centralized system was crucial for gaining product-wide visibility and control.
2. Automated SBOM Generation: By integrating the asset inventory with the centralized scan platform, Splunk achieved the ability to generate SBOMs quickly and easily. This process involves mapping product data from the inventory with scan results from various tools, then formatting the output into a standardized SBOM document, meeting compliance requirements and enhancing supply chain transparency.
3. VEX is Essential for Actionability: The most significant finding is the critical role of VEX in addressing the "flood of vulnerabilities." Splunk's data showed that over 26% of reported vulnerabilities were not exploitable or actionable. VEX statements provide the necessary context to filter these out, allowing development teams to focus on high-priority, genuinely exploitable issues, thereby drastically improving developer experience and reducing security debt.
4. Leveraging Existing Issue Tracking Systems for VEX Data Collection: Instead of building a new tool or buying an external solution, Splunk successfully repurposed their existing issue tracking systems as an abstraction layer for collecting VEX information. This approach significantly minimized the learning curve for developers, integrated seamlessly into existing workflows, and provided a mandated mechanism for collecting essential exploitability data (e.g., developers cannot close an issue without setting VEX details). This practical solution offers a high-impact, low-friction pathway for VEX adoption.
5. Context Building Enables Smart Remediation Decisions: By collecting remediation data through the VEX-integrated issue tracker, Splunk can build rich context around packages and issues. This context, derived from both external sources (e.g., public vulnerability databases, availability of fixes) and internal institutional knowledge (e.g., patterns of false positives, component usage), empowers automated decision-making. For instance, issues can be automatically closed if no fix is available or if they are repeatedly identified as false positives based on established internal patterns.
6. Smart Insights Drive Collaboration and Efficiency: The centralized data collection and analysis enable smart insights. The Security Central dashboard provides a bird's-eye view of security issues, allowing security teams to identify common vulnerabilities across product teams. This fosters collaboration by alerting teams when others are working on similar issues, preventing redundant efforts and promoting knowledge sharing across silos.

These findings collectively illustrate a strategic shift from simply detecting vulnerabilities to intelligently managing their exploitability and impact, driven by data and integrated workflows.

Technical Deep Dive

▶ Watch: Overview of the new centralized scanning system architecture (4:50)

Splunk's technical solution evolved from a decentralized "shift-left" model in 2022 to a sophisticated, centralized system designed to manage vulnerabilities at scale.

Initial Shift-Left Workflow (2022):

The original setup involved developers pushing code to a central repository, triggering scans via CI/CD pipelines. Scan results were sent to product security teams for review, and remediation feedback was then provided to developers. This process was developer-friendly, allowing customization of scan configurations. However, it lacked a consolidated view of vulnerabilities across all products, making it difficult for product security to gather product-wide vulnerability information.

Transition to a Centralized Model:

To address the lack of collective data and capitalize on the US Executive Order for SBOMs, Splunk transitioned to a centralized asset inventory and scan platform.

1. Asset Inventory Management:

• Goal: To map code repositories and build artifacts to specific products, providing a comprehensive overview.
• Implementation: Splunk developed internal tooling, such as the CTS portal, to collect this information. Developers use this portal to onboard or offboard their code repositories and artifacts for scanning.
• Data Collected: Beyond basic repository information, the portal collects details like ticket assignment owners, preferred branches for scans, and specific CVEs of interest for which tickets should be created.
• Benefits: Scalability (easily add any number of repositories/artifacts) and crucial product-level telemetry.
• Challenges: Ongoing operational costs and the continuous effort required to keep the inventory data up-to-date.

1. Centralized Scan Platform:

• Goal: Build a user-friendly system for end-to-end scanning operations, minimizing product team involvement.
• Workflow:
• A main program retrieves product data from the asset inventory.
• It requests the central scan platform to initiate scans on specified repositories.
• The scan platform uses various security tools (e.g., SAST, SCA, threat modeling, credential management tools) to perform scans.
• Scan results are retrieved from the tools and sent back to the main program.
• The main program processes these results, maps them to the correct owners, and stores the data in a central storage.
• Benefits: Automation of the entire scanning lifecycle, from initiation to ticket creation and notification.

1. SBOM Generator:

• A direct advantage of the centralized system was the ability to generate SBOMs quickly and easily.
• The system combines product data from the asset inventory with scan results from the tools.
• This data is mapped and formatted to produce standardized SBOM documents, ready for distribution or compliance purposes.

1. Security Central Dashboard:

• This is a key output of the centralized system, providing a dashboard of metrics and insights.
• Metrics: Distribution of findings by scan type (SAST, OSS, etc.) and severity, count of tickets created per type.
• Detailed Views: Filterable lists of tickets based on assignee, severity, product, or employee.
• Trends and Patterns: Visualizations of average ticket closure times over 3 months, distribution of findings by severity, and top 3 organization-wide product and infrastructure vulnerabilities.
• CVE Filtering: A crucial feature allows users to quickly search for a specific CVE (e.g., Log4j) and identify all products using the associated vulnerable component. This capability was only possible due to the integrated asset inventory and centralized scan data.

Integrating VEX for Exploitability Determination:

The technical solution then extends to address the core problem of non-actionable vulnerabilities using VEX.

1. Understanding Exploitability:

• Definition: The likelihood of a vulnerability being exploited.
• Methods:
• Consulting public sources (databases, advisories).
• Performing reachability analysis (determining if the vulnerable code path is actually used by the product).
• Developer feedback (the focus of Splunk's VEX implementation).

1. VEX Statement:

• Purpose: To indicate the state of a software product or component concerning a vulnerability, communicating exploitability information.
• Key Attributes: Machine-readable, complements SBOMs (can be embedded), supports automation and standardization of vulnerability communication.
• Recommended Statuses (Splunk's Usage): Not Affected, Affected, Fixed, and Under Investigation.

1. Challenges in VEX Generation:

• Tracking remediation data and linking it back to the product inventory.
• Mandating the collection of VEX information.
• The necessity of developer feedback, as tools alone generate false positives/negatives, and developers best understand their build systems and component usage.

1. Solution: Leveraging Issue Tracking Systems:

• Strategy: Utilize existing issue tracking systems as an abstraction layer over the security scanning suite.
• Advantages:
• Abstraction: Developers interact with a single system, not multiple scanning tools.
• Mandate: Issues cannot be closed without setting VEX details, ensuring data collection.
• Minimized Learning Curve: Teams already familiar with issue trackers.
• Standardization: A generic issue template ensures consistent data collection.
• Clear Exploitability Answer: Direct feedback from developers.
• Developer Ownership & Audit Trail: Tracks who owns remediation and provides historical context.
• Component Usage Insights: Understanding how components are used in practice.

1. Automated VEX Generation:

• The issue tracker collects remediation information (including VEX status).
• This data is stored in a dedicated database table.
• The system correlates this remediation data with the product inventory.
• The final output is VEX-embedded SBOMs, providing a complete picture of components and their exploitability status.

Building Context and Smart Insights:

With remediation data flowing in, Splunk began building context around packages and issues to automate decisions.

• External Sources: When an issue arises, the system checks public sources. If no fix is available, the issue can be automatically closed.
• Internal Sources (Institutional Knowledge):
• Remediation data helps identify patterns. If an issue is seen multiple times and consistently marked as a false positive (with reliable internal validation), future occurrences can be automatically closed, reducing "debt."
• This institutional knowledge is tool-agnostic, making the system scalable even if new scanning tools are integrated.
• The data also enables the generation of dependency graphs, further enriching context.
• Smart Insights:
• The centralized data provides product security with a "bird's-eye view."
• It identifies common CVEs impacting multiple product teams.
• Insights are provided to teams, encouraging them to collaborate and share remediation strategies, preventing siloed efforts.

This detailed technical architecture demonstrates Splunk's comprehensive approach to transforming vulnerability management from a reactive, alert-driven process into a proactive, intelligent, and context-aware security operation.

Demo / Proof of Concept

▶ Watch: Automated SBOM generation and Security Central dashboard (5:50)

While the talk did not feature a live coding demonstration or a traditional "proof of concept" in the sense of exploiting a vulnerability, the speakers effectively demonstrated the capabilities and practical application of their implemented system through detailed explanations and visual representations of their Security Central dashboard.

The Security Central dashboard serves as the nerve center for their centralized vulnerability management system, showcasing the aggregated data and insights derived from their asset inventory and scan platform. Key features demonstrated include:

• Distribution of Findings: Visualizations showing the breakdown of vulnerabilities by scan type (SAST, OSS, threat modeling, credential management) and severity.
• Ticket Counts: Displaying the number of tickets created for each scan type, providing an overview of workload and remediation progress.
• Detailed Filtering: The ability to filter lists of tickets based on various criteria such as assignee, severity, product, or employee, allowing security teams to quickly drill down into specific areas of concern.
• Trends and Patterns: Graphs illustrating historical data, such as the average time taken to close tickets over a three-month period, and the distribution of findings by severity over time.
• Top Vulnerabilities: Dashboards highlighting the top three product-wide and infrastructure-wide vulnerabilities across the organization, providing strategic insights into common risks.
• CVE Lookup Feature: A particularly impactful demonstration was the ability to quickly search for a specific CVE, such as Log4j, within the Security Central dashboard. This feature immediately reveals which products are utilizing the vulnerable component associated with that CVE, enabling rapid identification of affected systems and targeted response efforts. This capability directly showcases the power of having a centralized asset inventory and integrated scan results.

Through these dashboard views and feature descriptions, Splunk effectively demonstrated how their custom-built system provides actionable intelligence, improves visibility, and facilitates efficient vulnerability management, serving as a practical proof of the concepts discussed.

Defensive Implications

▶ Watch: Practical application: Filtering products by specific CVEs like Log4j (7:45)

The detailed approach presented by Splunk offers several crucial defensive implications for organizations striving to improve their security posture and manage open-source risks effectively:

1. Prioritize Centralized Asset Inventory: Defenders must establish and maintain a centralized asset inventory that comprehensively maps code repositories, build artifacts, and deployed products. This is the foundational step for gaining complete visibility into the software supply chain and enabling product-level telemetry, essential for any robust vulnerability management program.
2. Implement Centralized Scanning: Consolidate security scanning (SCA, SAST, DAST, etc.) into a centralized platform. This ensures consistent application of security policies, reduces operational overhead for product teams, and provides a unified data source for vulnerability information, feeding into dashboards like Splunk's Security Central.
3. Embrace SBOM Generation: Actively generate and utilize SBOMs for all software products. This not only aids in compliance with mandates (like the US Executive Order) but also provides a machine-readable list of components, making it easier to identify exposure to newly disclosed vulnerabilities (e.g., quickly finding all products affected by a Log4j-like event).
4. Adopt VEX to Combat Alert Fatigue: Implement Vulnerability Exploitability eXchange (VEX) as a core component of the vulnerability management process. Defenders should integrate VEX statements (Not Affected, Affected, Fixed, Under Investigation) to filter out non-exploitable or non-actionable vulnerabilities. This dramatically reduces alert fatigue for developers and security teams, allowing them to focus resources on genuine risks.
5. Leverage Existing Tools for VEX Data Collection: Instead of investing in new, potentially disruptive tools, organizations should explore how existing issue tracking systems can be adapted to collect VEX information. By making VEX status a mandatory field for closing vulnerability tickets, defenders can seamlessly integrate this crucial data into developers' existing workflows with minimal learning curve.
6. Build Institutional Knowledge and Context: Develop mechanisms to build and codify context around vulnerabilities and packages. This involves analyzing remediation data from internal sources (e.g., historical false positives, component usage patterns) and integrating external intelligence (e.g., public vulnerability databases, availability of fixes). This institutional knowledge enables smarter, automated decisions, such as automatically closing issues with no available fix or those consistently proven to be false positives.
7. Foster Collaboration with Smart Insights: Utilize centralized data to generate smart insights and identify common vulnerabilities across different product teams. Defenders should actively facilitate communication and collaboration between teams facing similar issues, sharing remediation strategies and preventing redundant efforts. This shifts the security team's role from gatekeepers to enablers of secure development.
8. Prepare for Future Automation with AI: Consider the future integration of agentic AI for advanced exploitability determination. By feeding historical remediation data and contextual information into AI models, organizations can potentially achieve highly accurate and automated assessments of whether a component is exploitable in their specific product environment.
9. Strive for Tool Agnosticism: Design internal systems (like the context-building engine) to be tool-agnostic. This ensures scalability and resilience, allowing organizations to integrate new scanning tools or replace existing ones without re-architecting their entire vulnerability remediation logic.

By adopting these defensive strategies, organizations can move beyond simply identifying vulnerabilities to intelligently managing their true risk, improving developer productivity, and building a more resilient software supply chain.

Key Takeaways

• Centralized Infrastructure is Paramount: A robust asset inventory combined with a centralized scan platform is the foundational requirement for effective vulnerability management, enabling comprehensive visibility, automated SBOM generation, and product-level telemetry.
• VEX is a Game-Changer for Actionability: Vulnerability Exploitability eXchange (VEX) is critical for filtering the "flood of vulnerabilities." By providing context on actual exploitability, VEX reduces false positives and non-actionable issues (over 26% in Splunk's case), significantly improving developer experience and focusing remediation efforts on genuine risks.
• Leverage Existing Workflows for VEX Adoption: Integrating VEX data collection into familiar tools, such as existing issue tracking systems, is a highly effective, low-friction strategy. This approach minimizes learning curves for developers and mandates the collection of essential exploitability information.
• Contextual Data Drives Intelligent Decisions: Building rich context from both internal remediation data and external sources allows for automated decision-making. This institutional knowledge enables smart closure of issues (e.g., no fix available, consistent false positives) and helps reduce security debt.
• Smart Insights Foster Collaboration: Centralized data and analytics provide smart insights into common vulnerabilities across teams. This enables security teams to facilitate collaboration, share best practices, and prevent redundant work, transforming security from a siloed function into a collaborative endeavor.
• Future-Proofing with AI and Open Standards: The ongoing evolution towards agentic AI for exploitability determination and the commitment to open-sourcing internal tools (like the centralized scan system) highlight a forward-looking approach to security challenges, aiming to benefit the wider industry.

About the Speaker(s)

Shina and Anushia are both Security Software Engineers at Splunk. In their roles, they are actively involved in implementing and improving security practices within the organization, particularly focusing on software supply chain security, vulnerability management, and developer experience. Their work at Splunk includes developing and deploying the centralized scanning platform, asset inventory, SBOM generation, and VEX integration strategies discussed in this presentation. They are dedicated to finding practical solutions to complex security challenges, as evidenced by their innovative approach to leveraging existing tools and building institutional knowledge to enhance vulnerability exploitability determination.