Six Years in Review: Transforming Company Culture to Embrace Risk

Six Years in Review: Transforming Company Culture to Embrace Risk

Speakers: Ariel Shin

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=cQE1OqCpeI8

Overview

This talk, "Six Years in Review: Transforming Company Culture to Embrace Risk," delivered by Ariel Shin, Product Security Engineering Manager at Twilio, details a transformative journey in vulnerability management. Shin takes the audience through a six-year evolution of a single program that successfully shifted company culture from ignoring risk to actively embracing it. The core of this transformation lies in the implementation and continuous iteration of a Democratized Vulnerability Management (DVM) approach, which reassigns the primary responsibility and accountability for vulnerabilities from the security team to engineering teams.

The presentation outlines the challenges and successes encountered while scaling this program across different organizational sizes and cultures, specifically at Segment and later at Twilio after its acquisition. It highlights how DVM not only improved remediation timelines and overall security posture but also fostered a deeper integration of security into everyday engineering discussions and roadmaps. This talk is crucial for security professionals grappling with the perennial problem of scaling security efforts, managing ever-growing backlogs, and building a sustainable security culture within large, dynamic organizations.

Ariel Shin, with her background as a pentester and extensive experience in AppSec at 1 Medical, Segment, and Twilio, brings a practical, battle-tested perspective to the complexities of vulnerability management. Her insights into navigating leadership resistance, fostering engineering buy-in, and building a foundational culture of trust offer a valuable roadmap for organizations looking to move beyond traditional, often burnout-inducing, security models. The talk emphasizes that vulnerability management is not merely a ticketing system but a powerful tool for driving cultural change and empowering engineers to become proactive partners in security.

Background

▶ Watch: Introduction of Democratized Vulnerability Management (DVM) - shifting respon... (2:00)

The traditional landscape of vulnerability management, as described by Ariel Shin, is often characterized as a "noisy, never-ending ticketing system." Security teams are perpetually overwhelmed, with backlogs growing faster than they can be addressed. This leads to security engineers experiencing burnout from constantly chasing down engineers for breached Service Level Agreements (SLAs) and a general lack of investment from engineering teams. For many organizations, especially public companies, the pressure to properly disclose exploited vulnerabilities further underscores the critical need for an effective and scalable vulnerability management program.

The problem's roots lie in the centralized nature of traditional security models, where the security team is solely responsible for identifying, prioritizing, and remediating vulnerabilities. This creates a bottleneck, as security teams rarely have the resources or direct control over engineering roadmaps to enforce timely fixes. Engineers, on the other hand, may view security tickets as an external imposition, lacking direct ownership or incentive to prioritize them amidst their core development tasks.

The journey towards a democratized approach began at Segment in 2018. Segment, a company that grew from 300 to 450 employees, successfully launched its DVM program, which found significant success in engaging engineers and leaders in driving down risk. When Twilio acquired Segment in 2021, Ariel Shin observed a stark contrast in vulnerability management practices. Twilio's existing processes were complicated, with three different workflows depending on the identifying security engineer, making it difficult to measure breached SLAs or the overall state of vulnerabilities. Engineers at Twilio often ignored security vulnerabilities, indicating a weaker security culture compared to Segment's established DVM success. This disparity highlighted the urgent need for a more scalable and culturally integrated approach at Twilio, leading to the decision to adapt and launch DVM across the larger organization. The core belief was that democratizing vulnerability management was essential for scaling security effectively and transforming the company's approach to risk.

Key Findings

▶ Watch: Addressing leadership buy-in challenges at Twilio with a two-pronged approach... (8:00)

The central finding of Ariel Shin's six-year review is that Democratized Vulnerability Management (DVM) is an effective and scalable strategy for transforming company culture to embrace risk. This approach fundamentally shifts the responsibility and accountability for vulnerabilities from a centralized security team to individual engineering teams, empowering them to own and manage their product's security posture.

Key findings and contributions from the DVM program's evolution include:

• Scalability and Efficiency: DVM proved highly effective in scaling security efforts across organizations of varying sizes, from Segment's initial growth to Twilio's much larger and more complex environment. This was evidenced by the ability to rapidly respond to zero-day incidents (e.g., spinning up comprehensive dashboards for a 2023 zero-day on day one, a stark contrast to the manual efforts required for Log4j in late 2021) and a measurable improvement in vulnerability burn-down rates, with more tickets closed within SLA over time.
• Core Pillars of DVM: The initial successful implementation at Segment was built on four pillars:

1. Engineers Own Risk: Vulnerability tickets are assigned to "risk owners" (engineering or product managers) who are accountable for the risk of their product or service.
2. Self-Service Due Date Extensions: Engineers can request SLA extensions, requiring approval from their leadership (VP/director), thereby increasing leadership accountability.
3. Transparent Vulnerability Reporting: Tickets include detailed descriptions, severity justifications (e.g., P1 vs. P2), and clear remediation criteria, allowing engineers to understand and even negotiate severity.
4. Visible and Meaningful Metrics: Regular reporting on security debt and action items drives accountability within engineering organizations.

• Addressing Scale and Cultural Challenges: When DVM was introduced at Twilio, additional pillars were identified as crucial for success in a larger, more diverse organization:
• Leadership Buy-in: Essential for resource allocation, driving accountability, and building trust. Required extensive communication and adaptation of messaging to various stakeholders.
• Engineering Buy-in: Critical for adoption, fostered through early engagement, tailored dashboards, and the introduction of Business Unit CISOs (BISOs) to act as security representatives.
• Awareness: Overcoming "white noise" through diverse communication channels, including a "Road Show" to 32 business units, proved more effective than emails or Slack messages alone.
• Ownership: Tackling the challenge of ambiguous or stale ownership information by attributing ownership upwards and quantifying the cost of the problem, leading to engineering-driven solutions like automated ownership catalogs.
• Noise Reduction: Managing the overwhelming volume of findings by prioritizing critical and high-severity issues (P1s and P2s) to build "security muscle," improving signal-to-noise ratio, and adapting notification methods (e.g., digest emails).
• The Foundation of Trust: Beyond the technical and process pillars, the most critical finding is the necessity of building a culture of trust. Maximizing vulnerability remediation at all costs can erode trust and lead to burnout. Instead, empowering engineers with autonomy and trusting their good intentions, even if it means slower remediation initially, fosters a sustainable security culture where engineers proactively embrace risk.
• Broader Application: The success of DVM extended beyond just vulnerabilities. The approach is now being explored and adopted by other teams (e.g., risk, Site Reliability) to standardize the management of enterprise risks and post-incident action items, forming a "Resilience Operations" group to avoid competing for engineering time.

These findings collectively demonstrate that a well-implemented DVM program, supported by a strong culture of trust and continuous adaptation, can fundamentally transform an organization's security posture and cultural relationship with risk.

Technical Deep Dive

▶ Watch: Implementation of the BISO (Business Unit CISO) role and weekly operational t... (14:00)

The evolution of vulnerability management programs typically follows a progression, as outlined by Ariel Shin, moving from rudimentary tracking to sophisticated, democratized systems.

Stages of Vulnerability Management Program Evolution:

1. Spreadsheets: The initial, often chaotic, stage where data is tracked across multiple, inconsistent spreadsheets. This method is unscalable and lacks a single reporting format, serving as a temporary solution while focusing on more pressing issues.
2. Traditional Vulnerability Management (Single Ticketing System): As companies mature, they transition to a centralized ticketing system (e.g., Jira) to manage vulnerabilities. This provides a single source of truth but often leads to security engineers being "burnt out from chasing Engineers breached SLAs and a general lack of investment from engineering." The backlog continues to grow, making scaling a significant challenge.
3. Democratized Vulnerability Management (DVM): This stage is adopted when scaling becomes the number one priority. DVM shifts the responsibility and accountability for vulnerabilities from the security team to engineering teams, aiming to empower engineers to own risk.

The Four Pillars of Segment's DVM (Initial Implementation):

Segment's DVM program, launched in 2018, was built on four foundational pillars designed to scale security effectively:

1. Engineers Own Risk and are Accountable for Vulnerabilities:

• Vulnerability tickets are explicitly assigned to risk owners, who are typically engineering managers or product managers, rather than individual engineers. This ensures that accountability for the risk of a product or service resides at a leadership level within engineering.

1. Self-Service Due Date Extensions:

• Recognizing that engineering teams face resource constraints, DVM allows engineers to request due date extensions. The approval process for these extensions involves their engineering VP or director, depending on the severity. This mechanism ensures that leadership is directly involved and accountable for the risk associated with delayed remediation.

1. Transparency in Vulnerability Reporting:

• Vulnerability tickets are designed to be highly transparent. Beyond a basic description, they include a detailed severity description, explaining why a vulnerability was prioritized as a P1 instead of a P2 (e.g., critical vs. high). This clarity allows engineers to review, understand, and even negotiate the assigned severity.
• Crucially, tickets also provide explicit criteria for remediation, clarifying what engineers need to do to mark a ticket as fixed, reducing ambiguity and back-and-forth.

1. Visible and Meaningful Metrics:

• To drive accountability, DVM emphasizes reporting on key metrics. At Segment, this involved sending monthly emails to engineering leadership detailing the security debt within their organizations and outlining action items to reduce it. These metrics provide a clear, data-driven view of security posture and progress.

Challenges and Additional Pillars for Twilio's DVM (Adaptation and Expansion):

When DVM was launched at Twilio, a much larger organization with a different culture and numerous mergers and acquisitions, the initial four pillars "started to crack." This necessitated the addition of several new pillars to strengthen the program:

1. Leadership Buy-in:

• Challenges: Initial resistance from various security teams (Cloud Security, Risk, VM) due to perceived opinionated changes, lack of full understanding of Twilio's existing programs, and concerns about resource allocation. Other challenges included siloed teams, complex bureaucratic processes, high leadership turnover, and competing priorities.
• Solutions: A "two-pronged approach" involved leveraging high-level leadership support (e.g., the CISO from Segment) to influence other leaders. Simultaneously, the message was adapted to address specific concerns: Cloud Security wanted tactical workflows and pilot programs; the VM team sought a clear vision and policy alignment; the Risk team required evidence from Segment's success to build trust. This process involved creating "around 42 documents, three slide decks over three quarters" to secure buy-in.

1. Engineering Buy-in:

• Challenges: Resistance to change, comfort with existing processes, lack of direct incentives, and poor communication leading to confusion. Scaling issues meant direct engagement with individual engineers was no longer feasible.
• Solutions: Early and frequent input from engineers helped adapt the program. Tailored dashboards were created for different roles (individual engineers vs. leaders). The concept of a Business Unit CISO (BISO) was introduced, assigning a security representative to each engineering organization. BISOs lead weekly "operational trust review meetings" where engineering leads and security representatives discuss top risks, review dashboards, provide context on remediation timelines, and identify unowned tickets.

1. Awareness:

• Challenges: Complexity of the new process, logistical difficulties in reaching a large and diverse audience, and "white noise" from common communication channels like Slack and email.
• Solutions: While primers, Wiki articles, and demo recordings provided consistent information, the most effective method was a "Road Show" to all "32 business units at Twilio." These 10-minute presentations, delivered in person, leveraged Twilio's strong meeting culture and allowed for direct engagement and Q&A, building goodwill.

1. Ownership:

• Challenges: Ambiguous owners (multiple teams, deprecated services, reorgs), high turnover leading to stale information, and the difficulty of motivating individual engineering managers to solve a collective problem. Security teams were spending significant time manually triaging and finding owners.
• Solutions: The strategy was to "attribute ownership upwards" (e.g., to teams or services rather than individuals) for greater stability. Crucially, the "ownership problem" was quantified by showing engineering leaders the hours spent by security on finding owners and the resulting delays. Segment's prior work on a centralized software catalog with a Jira integration was leveraged, allowing ownership information to auto-populate and auto-refresh in vulnerability tickets based on asset information.

1. Noise:

• Challenges: Engineers were overwhelmed by the sheer volume of findings, often lacking context (false positives, duplicates), leading to fatigue and distrust in security alerts. The "law of diminishing returns" meant lower-severity issues were ignored after critical ones were addressed.
• Solutions: The primary solution was to reduce the volume of findings by prioritizing remediation of critical (P1) and high (P2) vulnerabilities first, building "Security Muscle." This allowed engineers to tackle manageable debt before expanding focus. Other solutions included improving the signal-to-noise ratio through fine-tuning tooling, defining ticket creation standards, and including a "score for Ticket quality." Notification methods were adapted (Slack for notifications, email for escalation, digest messages instead of individual alerts, dashboards for bulk action). Regular security hygiene activities and recognition programs (gamification) were also implemented.

The Foundation of Trust:

Ariel Shin emphasizes that regardless of the specific pillars, the ultimate foundation for any DVM program is a culture of trust. Maximizing vulnerability remediation at all costs can be "inherently at odds with giving folks the autonomy and responsibility to remove immediate vulnerabilities." Trusting engineers to make the right decisions, even if it means slower remediation, is crucial for long-term sustainability and preventing burnout. This culture fosters empowerment, values opinions, and ensures security engineers are confident that engineers will protect the company.

Resilience Operations:

The success of DVM has led to its expansion into a broader initiative called Resilience Operations. This involves uniting vulnerability management, risk, and Site Reliability (SR) teams to standardize how vulnerabilities, enterprise risks, and post-incident action items are prioritized and reported to engineering. The goal is to stop competing for engineering time and instead collaborate to resolve the most pressing organizational issues.

Demo / Proof of Concept

▶ Watch: Strategy for noise reduction: prioritizing critical and high vulnerabilities ... (25:50)

While the talk did not feature a live, interactive demonstration of the DVM system, Ariel Shin presented compelling evidence and metrics that serve as a proof of concept for the program's effectiveness. It's important to note the speaker's disclaimer that "metrics have been fictionalized due to lack of company approval," but are "very fitting" and "based off a true story."

The primary demonstrations of DVM's success were:

1. Zero-Day Incident Response Comparison:

• Log4j (late 2021): This incident occurred before DVM was fully launched at Twilio. The response was characterized by significant manual effort. The security team lacked an easy way to track all repositories, and many known repositories were in an "unknown state of remediation." This required "many, many manual hours" to confirm affected repositories across Twilio and its acquisitions, and to verify remediation status.
• A 2023 Zero-Day: In contrast, for a zero-day incident in 2023, the DVM program enabled the security team to "spin up a dashboard like this on day one." This dashboard provided comprehensive coverage of all affected assets at Twilio and immediately showed a "downward trend of affected assets as engineering buring down risk quickly and efficiently." This stark comparison powerfully illustrates the DVM program's ability to provide rapid visibility and drive efficient remediation during critical security events.

1. Vulnerability Burn-Down Chart:

• The speaker presented a conceptual burn-down chart for all vulnerabilities across the organization. She noted that "before we launched stvm a dashboard like this wasn't even possible."
• The chart depicted the program's initial months showing that "most tickets had breached and very few tickets were closed within SLA." However, as time progressed and the DVM program matured, the chart clearly showed "more tickets Closed within SLA and fewer and fewer tickets Closed outside of SLA." This visual representation served as a strong indicator of improved remediation efficiency and adherence to security timelines, demonstrating that engineers were actively engaging with and resolving vulnerabilities.

These examples, though presented conceptually due to data restrictions, effectively illustrate the tangible benefits of a democratized approach: enhanced visibility, faster response times, and a measurable improvement in overall security posture driven by engineering ownership.

Defensive Implications

▶ Watch: Demonstrating DVM success with metrics (Log4j response, burn-down charts) and... (30:00)

The insights from Twilio's six-year journey with Democratized Vulnerability Management offer several critical defensive implications for organizations aiming to strengthen their security posture and culture:

1. Shift Accountability to Engineering: The most significant defensive implication is the imperative to move away from a centralized security team being solely responsible for vulnerability remediation. By shifting accountability to engineering teams and their leadership (e.g., "risk owners"), organizations can scale security efforts more effectively, leverage engineering bandwidth, and foster a sense of direct ownership over product security.
2. Prioritize Leadership and Engineering Buy-in: Implementing a DVM program is not merely a technical change; it's a cultural transformation. Defenders must recognize that securing buy-in from both high-level leadership (CISO, VPs) and engineering managers is non-negotiable. This requires significant investment in communication, adapting messages to different stakeholders (tactical, visionary, evidence-based), and demonstrating tangible business outcomes. Without this foundational support, even the best-designed program will falter.
3. Build a Culture of Trust: A critical defensive strategy is to cultivate a culture where security trusts engineers to act with good intention and engineers feel empowered, not just policed. While metrics are important, defenders must balance the drive for rapid remediation with the need to build autonomy and respect. Over-aggressive "fire alarm" tactics can lead to burnout and distrust, ultimately weakening the security posture. Focus on long-term engagement over short-term compliance.
4. Automate and Centralize Ownership Information: Ambiguous or stale ownership is a major impediment to efficient remediation. Defenders should advocate for and invest in engineering-driven solutions, such as centralized software catalogs integrated with ticketing systems, to automatically attribute and refresh ownership information. This reduces the security team's manual triage burden and ensures vulnerabilities are routed to the correct teams swiftly.
5. Reduce Noise and Improve Signal-to-Noise Ratio: Overwhelming engineers with a high volume of findings, especially low-severity or false positives, leads to fatigue and ignored alerts. Defenders should strategically reduce noise by prioritizing critical and high-severity vulnerabilities (P1s and P2s) to build "security muscle" first. Additionally, efforts to fine-tune security tooling, define clear ticket quality standards, and adapt notification methods (e.g., digest emails, dashboards for bulk action) are crucial to ensure engineers focus on truly impactful issues.
6. Embed Security into Engineering Workflows: The DVM approach encourages the integration of security fixes into everyday engineering workflows and roadmaps. Defenders should facilitate this by providing transparent vulnerability reporting, clear remediation criteria, and dedicated security partners (like BISOs) who can sync with engineering leads on top risks and provide continuous feedback. This proactive integration reduces the likelihood of vulnerabilities becoming incidents.
7. Standardize Risk Management Across Teams: The expansion of DVM into "Resilience Operations" suggests a broader defensive strategy: uniting different risk-focused teams (vulnerability management, enterprise risk, incident response) to standardize how all types of risks and action items are prioritized and communicated to engineering. This prevents internal competition for engineering resources and presents a unified front, ensuring the most pressing organizational issues are addressed collaboratively.
8. Invest in Continuous Awareness and Education: A new process, regardless of its benefits, requires ongoing education. Defenders should plan for continuous awareness campaigns, adapting communication channels and formats (e.g., "Road Shows," tailored dashboards) to ensure all engineers understand the process, their roles, and the importance of their contributions to security.

By adopting these defensive implications, organizations can move beyond a reactive, bottlenecked security model to a proactive, culturally integrated approach where security is a shared responsibility, deeply embedded in the engineering DNA.

Key Takeaways

• Democratized Vulnerability Management (DVM) is a proven strategy for scaling security: By shifting accountability for vulnerabilities from the security team to engineering teams, DVM effectively distributes the workload and fosters greater ownership, leading to more efficient remediation.
• Leadership and engineering buy-in are non-negotiable prerequisites: Successful DVM implementation requires significant, sustained effort in securing support from all levels of leadership and engineering, adapting communication strategies to address diverse concerns, and building strong partnerships.
• Clear and automated ownership is critical for efficiency: Ambiguous or stale ownership information severely impedes remediation. Investing in systems that automatically attribute and refresh ownership, ideally by linking to a centralized software catalog, drastically improves the speed and accuracy of vulnerability assignment.
• Reducing "noise" is essential to prevent engineer fatigue: Overwhelming engineers with a high volume of findings, especially lower-severity ones, leads to burnout and ignored alerts. Prioritizing critical and high-severity issues first, improving the signal-to-noise ratio of security tooling, and adapting notification methods are crucial for sustained engagement.
• A foundational culture of trust is paramount for long-term success: While metrics and processes are important, the ultimate sustainability of DVM relies on trusting engineers to act with good intention and empowering them to own risk. Maximizing remediation at all costs can erode this trust, leading to a less effective and more adversarial security culture.
• The DVM framework can be extended to broader risk management: The principles of democratized ownership and transparent reporting can be applied beyond just vulnerabilities to manage enterprise risks and post-incident action items, fostering a unified "Resilience Operations" approach that optimizes engineering time across various security and reliability concerns.

About the Speaker(s)

Ariel Shin is a Product Security Engineering Manager at Twilio. Her career began as a pentester, providing her with a foundational understanding of offensive security. She then transitioned into Application Security (AppSec) roles, first at 1 Medical, and subsequently at Segment. Following Twilio's acquisition of Segment, Ariel joined Twilio, where she has been for almost three years. In her current role, she manages the product security reactive team, which focuses on securing the second half of the Software Development Life Cycle (SDLC) and preventing vulnerabilities from escalating into security incidents. Her extensive experience across different companies and security disciplines has provided her with unique insights into transforming company culture to embrace risk through innovative vulnerability management programs.