S4x24 Main Stage Interview With Rob Lee

S4x24 Main Stage Interview With Rob Lee

Speakers: Robert M. Lee

Conference: S4

YouTube: https://www.youtube.com/watch?v=l4c0nY5vk08

Overview

This S4x24 Main Stage interview features a retrospective conversation with Robert M. Lee, a prominent figure in the industrial control system (ICS) security community. The discussion, moderated by Dale Peterson, marks a decade since Lee's first appearance at S4, offering a unique opportunity to reflect on the significant shifts and persistent challenges within operational technology (OT) security. The talk delves into unexpected developments over the past ten years, both positive advancements in executive-level understanding of OT security and concerning trends regarding the misguided push for IT/OT convergence. A central theme revolves around the nuanced distinction between "deployed" and "employed" in the context of advanced persistent threat (APT) capabilities, particularly in reference to the notorious Pipe Dream (also known as Incontroller) malware.

The conversation underscores the critical importance of understanding the unique physics and operational requirements of industrial environments, advocating against a simplistic "just do IT" approach to OT security. Lee emphasizes the pervasive visibility problem within critical infrastructure networks, highlighting how a lack of comprehensive monitoring leaves organizations vulnerable to undetected and prolonged adversary presence. This interview serves as a crucial checkpoint for the ICS security community, prompting reflection on progress made, lessons learned, and the strategic direction required to safeguard the world's most critical systems against increasingly sophisticated threats.

Background

▶ Watch: Welcome and 10-year S4 history (0:00)

The S4x24 interview with Rob Lee is framed by a decade of evolution in the industrial control system (ICS) and operational technology (OT) security landscape. Ten years prior, Lee made his debut at S4, coinciding with the release of his book, "S.C.A.T.E. and Me." This historical context sets the stage for a discussion on how the field has transformed since. Historically, OT security was often an afterthought, subsumed under broader IT security initiatives or entirely overlooked due to a lack of specialized understanding. The unique characteristics of OT environments—such as legacy systems, real-time operational demands, proprietary protocols, and the direct impact on physical processes—often made traditional IT security solutions unsuitable or even detrimental.

A significant shift over the past decade, as highlighted by Lee, has been the increased awareness at the highest levels of government and corporate boards. What was once a niche concern for engineers and specialized security professionals has now ascended to the strategic agenda of presidents, parliamentary members, and corporate executives. This elevated understanding recognizes OT security as a fundamental component of national and economic security, particularly concerning critical infrastructure. However, this progress has been tempered by internal industry debates, with a persistent push from some quarters to simply consolidate OT security under existing IT frameworks, a move that Lee critically questions. The underlying problem, as always, is the inherent difference between informational technology (IT) and operational technology (OT), where the former prioritizes confidentiality, integrity, and availability, while the latter places availability and safety above all else, often with real-world physical consequences for failure. This foundational tension forms the backdrop for much of the ongoing discourse in the sector.

Key Findings

▶ Watch: Negative surprise: Internal community pushing 'just do IT' (1:20)

The interview reveals several key findings and observations from Rob Lee's decade-long perspective in the OT security domain:

1. Elevated Understanding of OT Security at Strategic Levels: A significant positive surprise for Lee has been the dramatic increase in understanding of OT security at the board and true government levels (e.g., presidents, parliamentary members). There is now a broader recognition that OT security is the "critical part of critical infrastructure" and a necessary area of focus, a development Lee did not anticipate ten years ago.
2. Persistent Internal Struggle for OT/IT Differentiation: Conversely, a negative surprise has been the continued internal pressure within the community to consolidate OT security under IT. Lee expresses concern about individuals advocating for a "just do IT" approach, arguing that this overlooks the fundamental differences rooted in physics and operations, and risks diluting the specialized focus required for OT environments. He warns against "pretend[ing] it's all T."
3. Nuance of "Deployed" vs. "Employed" in Threat Intelligence: The discussion around Pipe Dream (also known as Incontroller) highlighted a critical distinction in threat intelligence terminology. Lee clarifies that "deployed" means a capability exists somewhere (e.g., test site, research system, accessible location), while "employed" means it has been actively used against its intended target in a victim network. This distinction is crucial for assessing immediate threat levels.
4. Pipe Dream's "Left of Boom" Status: Regarding Pipe Dream, Lee confirms it was deployed somewhere in the world, which allowed intelligence agencies to gain access to it. However, crucially, it was not employed in any victim networks the adversary was targeting. This represents a "left of boom" moment, where the capability was discovered and potentially mitigated before it could cause operational disruption.
5. Adversary Target Identification: Despite not being employed, the adversary behind Pipe Dream had already "picked out key targets across the United States" where they intended to use the malware. This indicates a high level of preparation and strategic intent, even if the attack was interdicted.
6. The Pervasive Visibility Problem: Lee underscores the widespread lack of adequate visibility into OT networks. Citing the example of the Volt Typhoon report, which noted adversary persistence for 300 days, he points out that such reports often reflect the limits of an organization's log retention or monitoring capabilities, implying that the actual dwell time could be significantly longer. This "we don't know what we don't know" scenario highlights a fundamental challenge in detecting and responding to threats in OT environments.

Technical Deep Dive

▶ Watch: Clarifying 'employed' vs. 'deployed' for Pipe Dream (2:40)

While this S4x24 session is an interview reflecting on past events and broader strategic trends rather than a live technical presentation of a new exploit or architecture, the discussion touches upon critical technical concepts and their implications for OT security. The primary technical topic discussed is the Pipe Dream malware, also known as Incontroller.

Pipe Dream is recognized as one of the most sophisticated and disruptive malware toolkits ever discovered targeting industrial control systems. Its capabilities are designed to interact directly with and manipulate various industrial devices and protocols, making it a highly versatile and dangerous threat. Although this specific interview does not provide a detailed technical walkthrough of Pipe Dream's inner workings, Lee's comments offer crucial insights into its status and the broader context of its discovery.

Lee emphasizes the distinction between a threat being "deployed" versus "employed." When an ICS-specific malware like Pipe Dream is "deployed," it signifies that the adversary has developed the capability, potentially tested it, and has it ready for use. This means the malware exists in some accessible form, perhaps on a staging server, a research environment, or a system controlled by the threat actor. In the case of Pipe Dream, Lee states unequivocally that it was "deployed somewhere in the world." This deployment allowed various parties, presumably intelligence agencies and security researchers, to gain access to the toolkit. The ability to access and analyze the deployed malware before its operational use was a significant intelligence victory.

Conversely, "employed" means the malware has been actively launched and is operating within the target's network, attempting to achieve its malicious objective. Crucially, Lee confirms that Pipe Dream was not employed in any victim networks the adversary was targeting across the United States. This "left of boom" scenario means that the threat was identified and neutralized (or at least its immediate operational use was prevented) before it could cause disruption or damage to critical infrastructure. This highlights the effectiveness of intelligence-gathering and proactive defense.

The technical sophistication of Pipe Dream, even without a direct deep dive in this interview, implies its ability to:

• Target multiple industrial protocols: Reports on Pipe Dream have indicated its capability to interact with various widely used ICS protocols, such as Modbus TCP, OPC UA, and possibly others, allowing it to control devices from different vendors.
• Manipulate programmable logic controllers (PLCs): The malware is designed to directly interface with and reprogram PLCs, the digital computers that automate industrial processes, enabling it to disrupt, damage, or even destroy physical equipment.
• Modular architecture: Advanced ICS malware often employs a modular design, allowing components to be swapped or updated, enhancing its adaptability and making detection more challenging.

The discussion also indirectly touches upon the visibility problem in OT networks. Lee's reference to the Volt Typhoon report, where an adversary persisted for 300 days, underscores a fundamental technical challenge: many OT environments lack the comprehensive logging, monitoring, and network segmentation that would enable early detection of sophisticated threats. Without adequate visibility into network traffic, device configurations, and behavioral anomalies, even deployed capabilities can remain undetected for extended periods, making "left of boom" interdiction difficult. The technical implication is that organizations must invest in specialized OT network monitoring solutions, protocol analysis tools, and robust logging infrastructure to gain the necessary insights to detect and respond to threats effectively.

Demo / Proof of Concept

▶ Watch: Rob's lack of confidence in Pipe Dream visibility (3:50)

This S4x24 Main Stage interview with Rob Lee is a reflective discussion and not a technical presentation that would typically include a live demonstration or a proof of concept. The format is an interview, focusing on insights, experiences, and strategic reflections from Lee's extensive career in OT security, particularly over the last decade.

While the conversation references significant past events, such as the discovery of the Pipe Dream malware, it does not involve any live technical demonstrations of this or any other tool. Lee's comments about Pipe Dream are retrospective, discussing its "deployed but not employed" status and the intelligence community's efforts to interdict it before it could impact critical infrastructure. Therefore, this section serves to clarify that the interview's purpose was to share high-level perspectives and strategic takeaways rather than to showcase technical exploits or defensive mechanisms in action.

Defensive Implications

▶ Watch: The broader 'visibility problem' in industrial networks (4:10)

The insights shared by Rob Lee in this interview carry significant defensive implications for organizations operating industrial control systems and critical infrastructure. The discussion underscores the urgent need for a nuanced and specialized approach to OT security, moving beyond generic IT security practices.

1. Reject the "Just Do IT" Mentality: The most critical defensive implication is to firmly resist the temptation to simply consolidate OT security under IT frameworks without specialized consideration. Defenders must recognize that OT environments are fundamentally different due to their reliance on physics, real-time operations, proprietary protocols, and the direct impact on physical safety and processes. Implementing IT solutions verbatim in OT can lead to operational disruptions, safety hazards, and ineffective security. Instead, organizations should develop OT-specific security strategies, leveraging frameworks like NIST CSF for ICS or IEC 62443, which are tailored to industrial contexts.

1. Enhance OT Network Visibility: Lee's emphasis on the "visibility problem" is a call to action for every critical infrastructure owner and operator. Defenders cannot protect what they cannot see. This necessitates investments in deep packet inspection (DPI) for OT protocols, asset inventory solutions that accurately map all connected industrial devices (PLCs, RTUs, HMI, historians), and behavioral anomaly detection systems designed for industrial networks. Log retention policies also need to be extended well beyond typical IT durations, as adversaries can dwell in OT networks for months or even years, as suggested by reports like Volt Typhoon. Comprehensive visibility is the foundation for early detection and rapid response.

1. Understand "Left of Boom" Intelligence: The Pipe Dream case highlights the value of intelligence-driven defense. Organizations should strive to integrate threat intelligence specific to OT/ICS into their security operations. Understanding the capabilities, tactics, techniques, and procedures (TTPs) of adversaries targeting industrial systems (e.g., state-sponsored actors like those behind Pipe Dream) allows defenders to implement proactive measures. While direct access to classified intelligence like that on Pipe Dream is rare for most organizations, consuming publicly available threat intelligence reports from agencies like CISA, Mandiant, or Dragos (Lee's company) can provide actionable insights for hardening systems and detecting precursors to attacks.

1. Prioritize Operational Continuity and Safety: Unlike IT, where data confidentiality is often paramount, OT security prioritizes availability and safety. Defensive measures must be carefully vetted to ensure they do not disrupt critical operations or introduce new safety risks. This means thorough testing of all security controls in a non-production environment before deployment, close collaboration between IT and OT teams, and a deep understanding of the industrial processes being protected.

1. Develop Specialized OT Incident Response Capabilities: The unique nature of OT incidents (e.g., physical impact, specific forensic challenges, need for operational recovery) demands specialized incident response plans and teams. Defenders need to train personnel in handling OT-specific incidents, including understanding industrial protocols, device forensics, and the critical need to restore operations safely and efficiently. The "left of boom" scenario for Pipe Dream underscores that even if an attack is prevented, the potential for sophisticated, targeted capabilities requires robust readiness.

1. Foster a Culture of Continuous Learning and Adaptation: The rapid evolution of threats and technologies means that defenders must continuously learn and adapt. Staying informed about new malware, attack vectors, and defensive techniques through conferences like S4, industry groups, and specialized training is crucial. Rob Lee's reflection on a decade of change emphasizes that the OT security landscape is dynamic, and static defense strategies will inevitably fail.

Key Takeaways

• OT Security Maturity is Rising: High-level government bodies and corporate boards increasingly recognize OT security as a critical component of national infrastructure and business continuity, a significant positive shift over the past decade.
• Beware of IT/OT Convergence Pitfalls: A persistent and concerning trend is the push to apply generic IT security solutions to OT environments. This "just do IT" approach risks overlooking the unique physics, operational requirements, and safety considerations of industrial systems.
• "Deployed" vs. "Employed" Matters for Threat Assessment: Understanding the distinction between a threat capability being "deployed" (existing and ready) and "employed" (actively used against a target) is crucial for accurate threat intelligence and risk assessment, as exemplified by the Pipe Dream malware.
• Pipe Dream was "Left of Boom": The sophisticated Pipe Dream (Incontroller) malware was discovered and analyzed while "deployed" but before it was "employed" against its intended critical infrastructure targets in the U.S., representing a significant intelligence and defensive success.
• The Visibility Problem is Pervasive: A fundamental challenge in OT security remains the widespread lack of adequate network visibility, making it difficult for organizations to detect sophisticated adversaries who can dwell in networks for extended periods (e.g., 300+ days, as seen with Volt Typhoon).
• Specialized OT Defense is Non-Negotiable: Effective defense requires a dedicated focus on OT-specific security strategies, tools, and expertise, emphasizing operational availability and safety, and moving beyond a one-size-fits-all IT security mindset.

About the Speaker(s)

Robert M. Lee (often referred to as Rob Lee) is a highly respected and prominent figure in the field of industrial control system (ICS) security. The interview highlights his long-standing engagement with the S4 conference, marking a decade since his first appearance. Lee is known for his deep expertise in critical infrastructure protection, threat intelligence, and digital forensics specifically tailored for operational technology environments. While the transcript does not provide his specific title or company at the time of the S4x24 interview, his work and reputation are widely recognized within the cybersecurity community, particularly through his leadership at Dragos, Inc., a company dedicated to safeguarding industrial systems. His background includes service in the U.S. Air Force, where he focused on cyber warfare operations, contributing to his unique perspective on national security implications of ICS vulnerabilities. He is also an author, educator, and a frequent speaker at major security conferences, advocating for a pragmatic and specialized approach to securing the world's critical infrastructure.