TotalTest Simulations 2 Oh! From Exploits to Economics

TotalTest Simulations 2 Oh! From Exploits to Economics

Speakers: Nebu Varghese, Senior Director, FDI Consulting

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=2EYYmncELXs

Overview

In his DEF CON talk, "TotalTest Simulations 2 Oh! From Exploits to Economics," Nebu Varghese, a Senior Director in FDI Consulting's cybersecurity practice, presented a compelling framework designed to transform how organizations approach security testing. Moving beyond the limitations of traditional, one-off penetration tests and red team engagements, Varghese advocates for a continuous, data-driven methodology that not only identifies vulnerabilities but also quantifies their business impact and the return on security investment (ROSI).

The core premise of TotalTest Simulations is to bridge the "confidence gap" that exists between theoretical security controls and an organization's actual resilience against sophisticated attacks. Varghese argues that security should no longer be viewed merely as a cost center but as a strategic advantage, capable of protecting core business value. By integrating diverse security functions—from threat intelligence and red teaming to risk management and security operations—and translating technical outcomes into financial metrics, this approach empowers organizations to make informed, data-backed decisions about their cybersecurity posture and investments.

This talk is particularly relevant for CISOs, CFOs, and security leaders grappling with how to effectively measure and communicate the value of their cybersecurity programs to the board. It offers a practical blueprint for evolving security testing from a compliance-driven exercise into a dynamic, economically justifiable function that continuously improves an organization's defensive capabilities and overall cyber resilience.

Background

▶ Watch: Introduction: Shifting security from cost to strategic advantage (0:00)

The modern threat landscape is characterized by its relentless evolution and increasing sophistication. Nebu Varghese highlights that traditional security testing methodologies, such as annual penetration tests or red team exercises, have become "relics" in this dynamic environment. He likens a single annual red team to a "single blood sample" – providing a snapshot but failing to offer the continuous monitoring equivalent to a "heart rate monitor" needed for modern enterprise cyber health. This raises critical questions about the defensibility of such singular assessments and whether CISOs are simply "crossing their fingers" in hopes of avoiding a major incident.

Several factors contribute to this growing challenge. Organizations struggle to keep pace with rapid technological advancements, especially with the accelerated adoption of AI. Expertise and capabilities are often lacking, compounded by the prevalence of outdated and unpatched systems. Inadequate business processes, which may not align with adopted technologies and controls, further exacerbate the problem. A critical, often overlooked aspect is the human element, not just in terms of susceptibility to social engineering like phishing, but also in the coordination and clockwork-like operation of people and processes during a crisis. Siloed communications and a lack of collaboration among different departments (e.g., HR, PR, legal) during an incident further hinder effective response.

Historically, enterprise environments featured heterogeneous systems and unique configurations, making it harder for attackers who needed bespoke research and resources to build targeted attacks, as seen in early oil and gas sector incidents. However, Varghese points out a subtle but significant shift towards homogeneous environments, where standardization across common techniques, technologies, and vendors is prevalent. This standardization, while beneficial for ease of implementation and interoperability, inadvertently makes the attacker's job easier. Ransomware actors, for instance, can now more readily modify and port their attacks across diverse sectors and organizations, increasing the attack surface and potential impact. This homogeneity contributes to a confidence gap, where organizations feel secure because controls look good on paper, but their ability to truly withstand a coordinated, multi-stage attack remains unvalidated. Policies may be signed, but seamless, coordinated responses in a real crisis are often untested.

Existing red team objectives typically focus on technical aspects: attack surface reduction (identifying pathways from initial access to privileged access), security monitoring (improving rule sets and custom detection use cases), and defense in depth (tightening endpoint security, network architecture, and configurations). While these are crucial, Varghese argues that the "missing piece" is the engagement of stakeholder response, particularly leadership and business teams. Most tabletop scenarios are debated for their realism, but a red team removes that room for doubt, demonstrating actual breach potential. The TotalTest Simulations framework aims to integrate these critical business and leadership elements into the security testing cycle, ensuring a holistic and realistic validation of an organization's cyber resilience.

Key Findings

▶ Watch: Human element and siloed comms in crisis response (3:00)

The central finding of Nebu Varghese's talk is the inadequacy of traditional, standalone security testing and the imperative for a TotalTest Simulations approach. This framework represents a paradigm shift, moving from isolated technical assessments to continuous, integrated, and financially quantifiable security validation.

1. Agile, Objective-Based Testing Cycles: Instead of lengthy, annual red team engagements, TotalTest Simulations advocates for agile, objective-based cycles, typically lasting two to three weeks. These shorter, focused assessments target one or two key objectives, allowing for rapid iteration and continuous improvement. This approach ensures that security testing is not a marathon but a regular fitness check for the entire organization.

1. Integrated Cross-Functional Collaboration: A core finding is the necessity of breaking down organizational silos. TotalTest Simulations explicitly requires collaboration among various teams:

• Cyber Threat Intelligence (CTI): To build relevant, prioritized scenarios based on real-world threats.
• Red Team: To execute the simulated attacks.
• Risk and Compliance: To identify the organization's highest risks and ensure scenarios align with critical business impact.
• Vulnerability Management (VM) and Security Operations (SecOps): To ingest findings, improve existing processes, tune detection use cases, and feed back into CTI.
• Leadership and Business Teams (HR, PR, Legal): To test crisis response, communication, and coordination, moving beyond purely technical detection and response.

1. Quantifiable Metrics Beyond Technical Indicators: Varghese emphasizes that traditional metrics like Mean Time To Detect (MTTD) or Mean Time To Respond (MTTR), while valuable for blue teams, don't fully capture the business impact. The TotalTest framework introduces a suite of advanced, business-centric metrics:

• Meantime to Initial Access: How long it takes a red team to gain an initial foothold. An increasing time indicates stronger perimeter defenses.
• Meantime to Objectives: How long it takes the red team to achieve their defined objectives. A longer time signifies greater defensive efficacy.
• Meantime Undetected: The duration from initial compromise until the red team is detected.
• Exploitation Success Rate: The percentage of successful exploits.
• Lateral Movement Success Rate: The ease with which attackers can move within the network.
• Attack Surface Reduction Rates: A quantifiable measure of how effectively the organization is shrinking its exploitable surface.

1. Financial Quantification of Security Value: The most significant finding is the ability to translate security performance into financial terms, speaking the language of the C-suite. Key financial metrics include:

• Loss Per Incident (LPI) / Loss Per Simulation: A baseline financial value of potential loss for a given incident scenario, calculated by modeling fixed costs of a breach (downtime, data exfiltration) as a function of time the red team spends to achieve its goals. A quicker blue team response lowers this cost.
• Revenue Protected: By quantifying the reduction in LPI after implementing fixes, organizations can demonstrate the tangible revenue safeguarded by their security investments.
• Return on Security Investment (ROSI): This framework enables the calculation of a defensible ROSI. For example, showing that for every dollar spent on mitigations, an organization protects $1.92 in potential losses.

1. Benchmarking Against Industry Peers: The framework incorporates the ability to benchmark an organization's performance against industry averages. By leveraging data from reports like the Verizon DBIR and IBM Cost of Data Breach reports, combined with specific threat intelligence for the sector, organizations can establish an average cost for an incident and define an acceptable target value for incident costs, guiding strategic investments and cyber insurance conversations.

Technical Deep Dive

▶ Watch: Homogeneous environments simplify attacks for adversaries (4:00)

The TotalTest Simulations methodology extends far beyond conventional penetration testing, incorporating sophisticated techniques and data-driven approaches to simulate realistic attack scenarios and measure defensive efficacy.

At its core, the approach emphasizes agile, objective-based cycles rather than monolithic, annual engagements. These cycles typically span two to three weeks, focusing intensely on one or two key objectives. This allows for rapid feedback loops and continuous improvement, contrasting sharply with the "annual marathon" model.

A critical initial step involves attack path prioritization. This is where Cyber Threat Intelligence (CTI) plays a pivotal role, collaborating with Risk and Compliance teams. CTI provides insights into relevant threat actors, their Tactics, Techniques, and Procedures (TTPs), and common attack vectors for the organization's specific sector. Risk and Compliance, understanding the organization's highest risk assets and business processes, helps prioritize scenarios that would have the most significant impact. This ensures the red team focuses on "most valuable attack vectors" rather than operating "blind," maximizing the relevance and impact of each simulation. Internal red teams, possessing deep contextual knowledge of the organization's controls and existing gaps, can further refine these prioritized pathways within attack trees.

Varghese introduces the concept of fuzzing the assumptions of your entire security architecture. This goes beyond merely throwing packets at a server. It involves systematically challenging the underlying assumptions of:

• Access Management (AM) controls: How robust are permissions, authentication, and authorization?
• Network architecture: Are segmentation, firewall rules, and ingress/egress filtering truly effective?
• Golden images of systems: Are the base configurations of deployed systems secure, or do they harbor exploitable weaknesses?

The goal is to identify interdependencies and known weaknesses by rigorously testing how different security components interact and whether they collectively withstand an attack. This "fuzzing" helps organizations "break smarter" by uncovering critical oversights that simple vulnerability scanning might miss.

To construct novel multi-stage attack paths, the framework leverages principles from graph theory and data science. While tools already exist that use graph theory to map attack paths (e.g., BloodHound for Active Directory), Varghese emphasizes using data science to connect disparate data points and build more complex, realistic attack chains. This includes analyzing trust chains, not just within Active Directory forests and domains, but also considering vendor trust relationships. A backdoor in a vendor system, depending on threat intelligence, could be a critical attack vector, and the simulations must test these extended trust boundaries. Often, it's subtle misconfigurations that create the "low-hanging fruit" enabling attackers to achieve their end objectives.

The execution of these simulations often involves tools like Cobalt Strike for C2 (Command and Control) operations, which allows red teams to emulate sophisticated threat actors. A key technical enabler for quantifying outcomes is Red Elk. This tool is highlighted for its ability to pull and ingest logs automatically from C2 implants (like Cobalt Strike), enabling the calculation and documentation of various metrics. Red Elk and similar platforms help automate the data collection and analysis, which is crucial for the data-driven approach.

The continuous nature of TotalTest Simulations is facilitated by robust feedback loops:

• Vulnerability Management (VM): Findings from the red team, particularly critical vulnerabilities that led to footholds or privilege escalation, are fed back to VM teams. This helps them reprioritize fixes, addressing vulnerabilities that may have been deprioritized by automated scanners due to a lack of context on their exploitability in a multi-stage attack.
• Security Operations (SecOps): The blue team receives feedback on detection failures. This helps them tune existing Security Information and Event Management (SIEM) use cases, develop new custom detection rules, and improve their overall MITRE ATT&CK Framework coverage. For example, if a specific command execution went undetected, SecOps can refine rules to catch similar TTPs on critical systems.
• Cyber Threat Intelligence (CTI): Attack profiles and hypotheses used for threat hunts are recalibrated based on real-world simulation results. If a particular actor's TTPs were successfully emulated, CTI can refine their monitoring and intelligence gathering efforts.

By integrating these technical processes, the TotalTest Simulations framework ensures that security testing is not a standalone event but an integral, continuously improving component of the organization's defensive posture.

Demo / Proof of Concept

▶ Watch: The 'confidence gap': paper policies vs. real resilience (5:00)

While Nebu Varghese explicitly stated that he could not share live client data, his presentation included a conceptual demonstration of how the TotalTest methodology quantifies security effectiveness using illustrative figures and a clear financial model. This served as a powerful proof of concept for the underlying principles.

The conceptual demonstration began by introducing the specific metrics tracked during a TotalTest simulation cycle. These go beyond typical blue team metrics:

• Meantime to Initial Access (MTIA): This measures how long it takes the red team to gain an initial foothold. In the example, an initial MTIA might be relatively short. The goal is for this time to increase in subsequent cycles, indicating stronger perimeter defenses.
• Meantime to Objectives (MTO): This metric tracks the time taken for the red team to achieve their defined objectives (e.g., domain admin, data exfiltration). Again, an increase in MTO across cycles implies that the blue team is forcing the red team to work harder, which is a positive outcome.
• Meantime Undetected (MTU): This measures the duration from the red team's initial foothold until their activities are detected. A decreasing MTU is the desired outcome, showing improved detection capabilities.
• Exploitation Success Rate and Lateral Movement Success Rate: These indicate the efficiency of red team operations. A decreasing success rate for the red team signifies a more resilient environment.

The core of the financial proof of concept revolved around the Loss Per Incident (LPI), also referred to as Loss Per Simulation. The speaker explained that LPI is a function of time: the more time the red team spends to achieve its goals, the greater the potential loss. Conversely, the quicker the blue team detects and thwarts the red team, the lower the cost.

The conceptual demonstration presented a hypothetical scenario:

1. Baseline Simulation: An initial red team cycle is run. Metrics are recorded (e.g., MTIA, MTO, MTU). Financial modeling is performed to establish a baseline LPI. This involves making assumptions about the fixed costs of a breach, downtime, and data exfiltration, which businesses can derive from internal data or industry benchmarks. For instance, a baseline LPI might be calculated at $546,000.
2. Implementation of Fixes: Based on the findings of the first simulation, the organization implements mitigations (e.g., new processes, security products).
3. Rerun Simulation: A subsequent TotalTest cycle is conducted, and new metrics are collected.
4. Recalculation of LPI: The LPI is recalculated based on the improved performance. The green line in the speaker's conceptual graph showed a lowered LPI, demonstrating the impact of the fixes.

Using these recalculated figures, the presentation moved to quantifying Revenue Protected. For example, if the baseline LPI was $546,000 and, after improvements, the LPI was reduced, the difference represents the revenue protected. The example cited a $146,000 saving, derived from reductions in downtime and recovery costs due to improved detection and response times (e.g., Mean Time to Containment (TTC) reduced from four days to two days).

This $146,000 figure then formed the basis for calculating a tangible Return on Security Investment (ROSI). If the cost of implementing the mitigations was, for instance, $50,000, the organization could present to the board that for every dollar spent, they protected $1.92 in potential losses. This powerful, quantifiable data, derived from the simulation cycles, provides a defensible financial justification for security investments.

Finally, the conceptual demonstration showed how to benchmark these internal figures against industry data. By layering intelligence from reports like the Verizon DBIR and IBM Cost of Data Breach reports with sector-specific threat intelligence, organizations can establish an average industry cost for an incident. They can then compare their internal LPI against this benchmark and define a target acceptable cost (e.g., $40,000) to operate within their risk appetite. This provides a clear goal and demonstrates progress towards achieving industry-aligned security resilience.

This conceptual proof of concept, despite not using live client data, effectively illustrated the methodology's ability to translate complex technical security outcomes into clear, financially relevant metrics for the C-suite.

Defensive Implications

▶ Watch: Total Test: integrating leadership and business teams (6:00)

The TotalTest Simulations framework profoundly reshapes defensive strategies, moving organizations from a reactive, compliance-centric posture to a proactive, data-driven, and business-aligned approach. The implications for defenders are manifold and transformative:

1. Continuous Security Validation: The most significant implication is the shift from annual, point-in-time assessments to continuous security validation. Defenders can no longer rely on a single "blood sample" but must implement a "continuous heart rate monitor" for their cyber health. This means regularly testing, iterating, and improving defenses, ensuring they keep pace with the evolving threat landscape.

1. Integrated and Collaborative Defense: The framework mandates breaking down traditional silos within security teams and across the broader organization. CTI, red team, risk management, vulnerability management, and security operations (SecOps) must work in lockstep. This fosters a shared understanding of threats, risks, and defensive capabilities, leading to more coordinated and effective incident response. Furthermore, involving business teams (HR, PR, Legal) in crisis simulations ensures that the entire organization is prepared, not just the technical security staff.

1. Business-Aligned Security Strategy: Defenders gain the ability to articulate security value in the language of the business, particularly financial terms. By quantifying Loss Per Incident (LPI), Revenue Protected, and Return on Security Investment (ROSI), CISOs can directly demonstrate how security investments safeguard core business objectives. This empowers them to move beyond abstract risk discussions and secure the necessary budget and executive buy-in for critical security initiatives. It reframes security from a cost center to a strategic enabler.

1. Data-Driven Decision Making: The emphasis on specific, quantifiable metrics (e.g., meantime to initial access, exploitation success rate, attack surface reduction rates) provides defenders with objective data to measure progress. This allows for evidence-based decision-making, enabling teams to identify specific weaknesses, prioritize remediation efforts, and track the effectiveness of implemented controls. Tools like Red Elk facilitate the automated collection and analysis of this data, making the process scalable and consistent.

1. Enhanced Detection and Response Capabilities: The continuous feedback loop from red team simulations directly informs and improves blue team operations. SecOps teams can use red team findings to:

• Tune SIEM use cases: Refine existing rules and create new ones to detect specific TTPs that were successful during simulations.
• Improve MITRE ATT&CK coverage: Identify gaps in detection capabilities across the ATT&CK framework.
• Validate incident response playbooks: Test the efficacy of their processes, from initial alert to containment and eradication (time to eradicate).
• Refine threat hunting hypotheses: CTI teams can use real-world simulation data to recalibrate their understanding of relevant threats and focus their threat hunting efforts.

1. Proactive Vulnerability Management: The red team's ability to chain together subtle misconfigurations and seemingly low-priority vulnerabilities into critical attack paths provides invaluable context for vulnerability management teams. This helps them reprioritize remediation efforts, focusing on vulnerabilities that, while perhaps not "critical" in isolation, are highly exploitable in a multi-stage attack scenario.

1. Benchmarking and Risk Appetite Management: By integrating industry benchmark data (e.g., from Verizon DBIR, IBM Cost of Data Breach reports), defenders can understand their organization's security posture relative to peers. This informs discussions around risk appetite and tolerance levels, allowing organizations to set realistic goals for their security programs and demonstrate progress towards operating within acceptable thresholds. This data also strengthens conversations with cyber insurance providers, potentially leading to more favorable premiums.

1. Fuzzing Architecture Assumptions: The concept of "fuzzing" security architecture assumptions (AM controls, network design, golden images) encourages defenders to systematically challenge their own security designs. This proactive questioning of established controls helps uncover deep-seated architectural weaknesses and interdependencies that might otherwise remain hidden until exploited in a real attack.

In essence, TotalTest Simulations empowers defenders to move beyond simply "finding vulnerabilities" to strategically building and continuously validating an adaptive, resilient, and economically justifiable defensive ecosystem.

Key Takeaways

• Traditional security testing (annual pentests/red teams) is outdated. Organizations need to transition to continuous, agile, and objective-based security simulation cycles to keep pace with evolving threats and homogeneous IT environments.
• Integrate all security functions and business teams for holistic validation. Effective security testing requires seamless collaboration between CTI, red team, risk/compliance, vulnerability management, SecOps, and even leadership/business units (HR, PR) to test both technical controls and crisis response.
• Measure security effectiveness with business-relevant financial metrics. Move beyond technical-only metrics like MTTD/MTTR to quantify Loss Per Incident (LPI), Revenue Protected, and Return on Security Investment (ROSI) to speak the language of the C-suite and justify security investments.
• Leverage data science and automation for continuous improvement. Utilize tools like Red Elk to automatically ingest logs, calculate metrics, and track trends, providing objective data for decision-making and continuous feedback loops to improve detection, response, and vulnerability management.
• Proactively "fuzz" security architecture assumptions to uncover critical oversights. Beyond simple vulnerability scanning, systematically challenge the efficacy of access management controls, network architecture, and golden images to identify interdependencies and subtle misconfigurations that enable multi-stage attacks.
• Benchmark against industry data to contextualize performance and manage risk appetite. Use reports like Verizon DBIR and IBM Cost of Data Breach to compare your organization's LPI against industry averages, set target acceptable costs, and inform cyber insurance conversations.

About the Speaker(s)

Nebu Varghese is a Senior Director in the cybersecurity practice at FDI Consulting, based in London. With over 13 years of experience in the offensive security space, he specializes in penetration testing, red teaming, and other advanced security testing methodologies. Varghese holds a Master's degree in Software and System Security from the University of Oxford, and much of the TotalTest Simulations framework presented in this talk is inspired by his research during his studies. He is actively involved in community initiatives, particularly with the UK National Cyber Security Center (NCSC) where he contributes to security testing in the Operational Technology (OT) sector. Additionally, he occasionally participates in the red teaming chapter for FIRST (Forum of Incident Response and Security Teams), further demonstrating his commitment to advancing cybersecurity practices.