From Auditions to Opening Night: Selecting Security Tools that hits the high notes

From Auditions to Opening Night: Selecting Security Tools that hits the high notes

Speakers: Saurabh Sharma, Founding Security Engineer, Lime

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=Urusx0aZh1I

Overview

In the fast-paced world of cybersecurity, the allure of cutting-edge tools and zero-day exploits often overshadows a fundamental challenge: the successful selection and deployment of security solutions. Saurabh Sharma, a Founding Security Engineer at Lime, tackles this critical yet often-neglected topic in his BSides SF talk, "From Auditions to Opening Night: Selecting Security Tools that hits the high notes." Sharma candidly addresses the "security tools tragedy" – the common scenario where flashy vendor demos lead to expensive contracts, only for tools to languish unused, generating false positives, and draining team resources without delivering tangible value.

Sharma's presentation shifts the focus from the technical intricacies of attacks to the practical, human-centric process of integrating security tools into an organization. He argues that true security success hinges not just on identifying threats, but on effectively convincing stakeholders, aligning solutions with business outcomes, and ensuring seamless adoption by engineering teams. His talk provides a structured framework, drawing parallels to writing a song, to navigate the complex landscape of vendor pitches, pilot programs, and long-term tool sustenance, all while maintaining credibility and sanity within the security team.

This article delves into Sharma's experiences, highlighting the methodologies he developed to avoid common pitfalls and ensure that security investments genuinely enhance an organization's defensive posture. It's a deep dive into the strategic and tactical considerations necessary for any security professional looking to make impactful technology choices that resonate throughout their company, moving from a reactive stance to a proactive, integrated security culture.

Background

▶ Watch: Avoiding the 'security tools tragedy' in tool deployment (2:15)

The pervasive problem Sharma addresses stems from a disconnect between the perception of security tools and their operational reality. Security conferences frequently feature discussions on advanced attack techniques, novel detections, and complex threat landscapes. While these topics are vital, they often overshadow the foundational work required to effectively implement security measures. The "security tools tragedy" described by Sharma is a cycle: a vendor presents an exciting demo, a security team initiates a pilot, only to face a barrage of complaints about false positives, a security team drowning in configuration changes rather than actual security work, and leadership questioning the return on investment for six-figure contracts on tools that nobody uses.

Sharma illustrates this problem with concrete examples from his experience as a founding security engineer at a fast-growing startup, Lime. He notes that in an environment with limited security personnel and time, the challenge is not just technical, but also about convincing people of the value of security initiatives. Prior to implementing new solutions, the organization faced a grim reality: an ever-growing number of vulnerabilities in their software supply chain. Specifically, Sharma presented data showing approximately 6,000 vulnerabilities, with only 13 critical or high vulnerabilities mitigated in a calendar year out of thousands. This translated to an abysmal mean time to remediate (MTTR) of close to a calendar year (325 days), indicating a deeply reactive and ineffective security posture. Furthermore, none of the company's repositories had zero alerts, suggesting a systemic problem across different tech stacks. This baseline data underscored the urgent need to transition from a reactive state, where vulnerabilities were addressed long after they reached production, to a proactive one, preventing issues before they materialized. The core problem was not a lack of tools, but a lack of effective tool selection and deployment that could genuinely shift the organizational security culture.

Key Findings

▶ Watch: Real-world supply chain vulnerability problem data (4:40)

Sharma's core contribution is a practical framework for selecting and deploying security tools successfully, built on the principle of aligning security initiatives with business outcomes and effectively communicating value to diverse stakeholders. He emphasizes that before engaging with any vendor or seeing a single demo, one must first clearly define the problem being solved and the framework gaps it addresses (e.g., GDPR compliance needs).

To effectively pitch security solutions and gain organizational buy-in, Sharma proposes a "song" analogy, comprising four key elements:

1. The Hook: A single, concise sentence that condenses the problem without mentioning a solution. Its goal is to grab attention and make people "lean forward." For instance, for supply chain security, the hook was: "We are resolving 13 critical vulnerabilities in a calendar year out of thousands, and our mean time to remediate is close to a calendar year."
2. The Chorus: The data that backs up the hook, illustrating the pain points with baseline numbers and showing trends (e.g., open alerts trending up). This reiterates the problem until it sticks, focusing on quantitative evidence.
3. The Bridge: This connects the problem to the solution, explaining how a tool delivers outcomes that stakeholders care about, rather than just listing features. For example, instead of "reachability analysis," the outcome is "reducing manual workload by identifying the 5 exploitable vulnerabilities out of 200."
4. The Finale: The clearly defined success criteria for the tool's implementation. This acts as a "pseudo-contract," holding the security team accountable for proving the initial hypothesis and demonstrating the tool's value post-pilot.

Sharma applied this framework to two distinct case studies: a runtime security tool and a supply chain security tool. For runtime security, the problem was environmental drift and a lack of runtime context for vulnerabilities, leading to blind prioritization based solely on CVSS scores. The desired outcome was the reduction of manual triage workload through reachability analysis and enabling cloud security best practices. For the supply chain tool, the goal was a culture shift from reactive to proactive security, addressing the abysmal MTTR and blocking malicious packages. The key outcome here was integrating security feedback directly into the developer workflow (PRs, browser extensions) to empower developers to make secure decisions earlier. Across both scenarios, the emphasis was on data-driven problem definition, outcome-focused solution pitching, and clearly measurable success.

Technical Deep Dive

▶ Watch: Explaining security risk to diverse stakeholders (7:00)

Sharma detailed the application of his framework to two practical scenarios, revealing the technical challenges and the proposed solutions.

Runtime Security Tool

The primary problem identified for runtime security was environmental drift. Security controls and mitigations that worked perfectly in staging environments often failed or behaved unpredictably in production due to subtle environmental differences. This created a lack of confidence in the effectiveness of security measures once deployed. Compounding this, the security team suffered from a blind spot regarding which known vulnerabilities were actually exploitable in their specific production context. They were relying heavily on CVSS scores for prioritization, a method Sharma critically described as "horoscopes for vulnerabilities" – a high CVSS score doesn't necessarily mean exploitable, and exploitable doesn't mean exploited. Without runtime context, the team was "flying blind."

The chosen runtime security tool aimed to address this through several technical capabilities:

• Reachability Analysis: This was a crucial technical feature that translated directly into a key outcome. The tool could analyze the runtime environment to determine if a vulnerable component was actually accessible and exploitable. Sharma stated that often, less than 5% (and sometimes as low as 1-2%) of critical and high vulnerabilities are truly exploitable in a given environment. By identifying these critical few, the tool significantly reduced the manual workload of triaging individual vulnerabilities.
• Cloud Security Best Practices: The tool provided a mechanism to feature gate security-related best practices. This ensured that configurations and security policies applied in staging environments would behave identically and effectively in production, mitigating the environmental drift problem.
• Real-time Workload Monitoring for Zero Days: The ultimate technical goal and success criterion for the pilot was the ability to achieve real-time workload monitoring to detect and respond to zero-day exploits. This moved the organization towards a proactive stance against unknown threats.

Supply Chain Security Tool

For the supply chain security initiative, the technical problem was deeply intertwined with a cultural one: moving from a reactive state of fixing vulnerabilities post-deployment to a proactive state where security issues were prevented earlier in the development lifecycle. The existing state was characterized by the aforementioned 325-day MTTR for vulnerabilities and a complete inability to detect malicious packages or typosquatting attacks before they reached production. The current tools merely checked a static vulnerability database after a package was in use, offering no preventive capability.

The new supply chain security tool focused on integrating security into the developer workflow and providing early signals:

• SLSA Threat Mapping and Blocking: The tool was designed to map SLSA (Supply-chain Levels for Software Artifacts) threats and actively block malicious packages, including those involved in typosquatting, before they were merged into production. This represented a fundamental shift from post-facto detection to pre-emptive prevention.
• Developer Workflow Integration: This was a critical technical and cultural selling point. Instead of forcing developers to consult external wikis or separate tools, the solution brought security feedback directly to where developers work:
• PR-level Feedback: A bot commented on Pull Requests (PRs), detailing changes in dependencies, new capabilities introduced, and the security score of newly added packages. This empowered both the PR author and reviewer to make informed, security-conscious decisions.
• Browser Extension: For developers actively searching for open-source libraries, a browser extension provided immediate security feedback. This "shifted left" security even further, allowing developers to identify problematic dependencies before incorporating them into their code, preventing them from even reaching the PR stage.
• CI Output: The tool also provided feedback as part of the Continuous Integration (CI) pipeline, ensuring automated checks and blocks if security policies were violated.
• Rapid Detection and Remediation: The technical finale for this pilot was to achieve a detection time of less than one day and a remediation time of less than one week for supply chain issues, a stark contrast to the previous 325-day MTTR.

In both cases, Sharma emphasized that the technical features were not presented in isolation but were directly linked to measurable outcomes and a significant reduction in risk and manual effort, making the value proposition clear to both engineering and leadership.

Demo / Proof of Concept

▶ Watch: 'Writing a song' framework for pitching security initiatives (7:50)

Sharma's approach to the pilot (Proof of Concept) phase is highly structured and strategic, designed to yield actionable data and secure genuine adoption. He recommends time-boxing pilots, with 30-60 days being a "sweet spot." This duration is long enough to gather meaningful data but short enough to maintain momentum and prevent stakeholder fatigue. Crucially, the pilot should be opt-in, not mandatory. Mandatory pilots often result in teams doing the "bare minimum" and withholding honest feedback, which is vital for product success. Instead, Sharma advises selecting 2-3 teams with diverse tech stacks and deployment methodologies to ensure broad applicability of findings.

The most critical element of the pilot is defining success criteria from day one. Without this, the pilot concludes with ambiguity, leaving everyone wondering, "So, did it work?" Sharma categorizes measurement into two buckets:

1. Technical Metrics: These directly demonstrate whether the solution is solving the problem. Examples include:

• False positive rate: A low rate is crucial to maintain trust.
• Detection time: How quickly a threat is identified.
• Remediation time: How much the time to fix issues is shortened.
• Specific to the runtime security pilot, the "detection time dropped within a day" was an "infinite improvement" from a baseline of zero detection.
• For supply chain, the goal was less than one-day detection and less than one-week remediation.

1. Adoption Metrics: Often overlooked, these measure user engagement and satisfaction, both qualitatively and quantitatively:

• Are developers looking at alerts? Are they acting on them?
• Did anyone turn off the tool?
• How often do developers reach out for help? This indicates the tool's intuitiveness.
• Qualitative feedback, such as complaints on Slack or requests to disable the tool, are "real signals" that shouldn't be ignored.

During the pilot, a centralized UI (command center) provided the security team with an overview of protected repositories, PRs analyzed, and anomalies detected. For the supply chain pilot, specific technical demonstrations included:

• PR Analysis: The tool analyzed PRs, showing a very low alert rate (e.g., "only one commit alerted" out of many analyzed), indicating low noise and high signal.
• Developer Feedback Loop: This was a key "demo" of the tool's integration. A bot commented on PRs detailing dependency changes, new capabilities, and security scores. This empowered both authors and reviewers.
• Browser Extension: A browser extension provided real-time security insights when developers searched for open-source libraries, demonstrating how security could be shifted "even more left."

Sharma also emphasized the importance of being willing to close a pilot if it's not working. If engineers hate the tool, false positive rates are too high, or adoption numbers don't hold steady, it's a success in evaluation to stop, rather than a failure to invest more time and money in a non-viable solution. The ultimate litmus test for a successful pilot is if "at least one team says that we cannot be able to live without this tool."

Defensive Implications

▶ Watch: Example of a compelling 'hook' for a security pitch (8:20)

Sharma's talk offers crucial insights for defenders beyond the initial tool selection, focusing on common pitfalls, successful rollout strategies, and long-term maintenance.

Avoiding Common Mistakes:

1. Configuration Mistakes: Starting with overly stringent policies "out of the gate" is a common error. Aggressive policies lead to a flood of false alarms, causing developers to quickly ignore the tool. Once trust is lost, it's "incredibly hard to get that trust back." Sharma advises starting with permissive policies and gradually tightening them as confidence in the tool's accuracy grows.
2. Integration Challenges: Vendor documentation often simplifies integration, but real-world environments are complex. Every team has unique tech stacks and custom pipeline configurations. Defenders must budget double or triple the estimated time for integrations and be prepared to "fiddle a lot" to make the tool work across diverse environments.
3. "Set It and Forget It" Trap: Launching a pilot and then neglecting it is a recipe for failure. A designated member of the security team must "babysit" the pilot, collecting weekly feedback, adjusting settings, and fine-tuning configurations based on direct input from pilot teams.
4. Testing in Staging vs. Production: Sharma highlights the mistake of spending too much time configuring a tool in a clean staging environment. He strongly recommends testing in the production environment, which contains "years of accumulated weirdness." A tool must prove its ability to work in the organization's actual "mess," rather than an idealized setting.

Strategic Rollout and Sustenance:

1. Staged Rollout (vs. "Big Bang"): Sharma strongly advises against "big bang" rollouts, where a tool is deployed to the entire organization at once. While tempting for leadership to show quick coverage numbers, these often lead to "noise and disruption," causing a "revolt" and leaving the organization in a worse state. Instead, he recommends a phased approach:

• Start with pilot teams who are already familiar and have credibility.
• Expand to teams with similar tech stacks.
• Finally, roll out to the rest of the organization.
• Each phase should be approximately two weeks, with tight feedback loops for continuous improvement.

1. Leveraging Early Adopters: To overcome skepticism, make early adopters your "sales people." When a peer team champions a tool, others are "much more willing to listen" than to security.
2. Continuous Monitoring and Tuning: Security tools are not "deployed and done." They require constant monitoring, tuning of policies, and adaptation to evolving environments, new teams, and changes in production.
3. Tight Feedback Loop with Vendors: Maintain an active partnership with the vendor, providing candid feedback on what works and what doesn't. Responsive vendors who incorporate suggestions into their roadmaps are "worth way more than just writing checks."
4. Market Awareness: The security landscape evolves rapidly, with tools potentially becoming "old in 12 to 18 months." Defenders should conduct smaller, periodic assessments (every 12-18 months) to evaluate whether problems have evolved and if new solutions are needed.

Ultimately, defensive success is measured not by the number of vulnerabilities found, but by the effective reduction of risk through developer adoption, rapid detection, and swift remediation.

Key Takeaways

• Start with Business Outcomes, Not Vendors: Always define the problem and align it with specific business outcomes and framework gaps before engaging with any vendor or product. Shopping without a clear list leads to ineffective purchases.
• Pitch with Data and Outcomes: Build proposals using a "song" structure (hook, chorus, bridge, finale). Use baseline data to highlight pain points with numbers, and present solutions in terms of the measurable business outcomes they achieve, not just features.
• Define Success Criteria Early: Clearly define what constitutes success (and failure) for a tool before spending any money or starting a pilot. This creates accountability and a clear path for evaluation.
• Measure What Truly Matters: Focus on metrics that demonstrate actual risk reduction, such as detection time, remediation time, and, most importantly, developer adoption, rather than simply counting vulnerabilities. A tool is worthless if nobody uses it.
• Pilot Strategically and Be Flexible: Conduct time-boxed, opt-in pilots with diverse teams. Be prepared to adjust configurations, collect continuous feedback, and be willing to abandon a tool if it doesn't meet defined success criteria or faces strong user resistance.
• Roll Out Incrementally and Sustain Efforts: Avoid "big bang" rollouts. Opt for phased deployments, leverage early adopters as champions, and recognize that security tools require continuous monitoring, tuning, and ongoing vendor partnership to remain effective.

About the Speaker(s)

Saurabh Sharma is a Founding Security Engineer at Lime. With 12 years of experience in the security industry, Saurabh dedicates his time to enhancing company security, often navigating challenges with limited resources and an abundance of vendor communications. He is passionate about moving beyond just identifying zero-day exploits and fancy detection logics, focusing instead on the crucial, often overlooked, aspect of successfully selecting, deploying, and demonstrating the value of security tools. In his free time, Saurabh enjoys outdoorsy activities, particularly mountaineering, which he believes helps develop the tenacity valuable for a security engineer.