Safeguarding the Industrial Frontier OT SOC & Incident Response

Safeguarding the Industrial Frontier: OT SOC & Incident Response

Speakers: Adam Robbie; Joe (details from transcript)

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=AXN3sTAr9R4

Overview

This talk delves into the critical and often overlooked realm of Operational Technology (OT) security, specifically focusing on the establishment and operation of an OT Security Operations Center (SOC) and effective incident response strategies. Given by Adam Robbie and a colleague identified as Joe during the discussion, the presentation exposes the stark realities and unique challenges faced when securing industrial control systems (ICS), SCADA, and other OT environments. Unlike the rapidly evolving IT landscape, OT systems are characterized by incredibly long lifespans, legacy hardware, and a foundational lack of security-centric design, making traditional IT security approaches largely ineffective or impractical.

The speakers underscore that the prevailing "secure by design" philosophy, while ideal, often collides with the 30-to-40-year operational lifespan of industrial equipment, much of which predates modern cybersecurity considerations. This disparity necessitates a shift towards "secure by operations," emphasizing robust detection and response capabilities through a specialized OT SOC. The talk is particularly vital for organizations grappling with the convergence of IT and OT networks, offering pragmatic advice on how to navigate the complexities of securing systems where the stakes involve not just business continuity but also human lives and critical infrastructure. It serves as a crucial wake-up call for security professionals to understand, adapt to, and adequately address the distinct threat landscape of the industrial frontier.

Background

▶ Watch: Legacy OT devices lack relevant security logs (0:00)

The genesis of the challenges discussed in this talk lies in the fundamental differences between Information Technology (IT) and Operational Technology (OT) environments. For decades, OT systems were designed for reliability, safety, and performance, often operating in air-gapped or isolated networks, with little to no consideration for external cyber threats. This historical context has resulted in a pervasive presence of legacy devices that remain operational for 30, 40, or even more years – a lifespan that often predates the very concept of security logging or modern network protocols. As one speaker humorously noted, some of these devices were installed when "Nixon was in office," long before "logging was invented."

A significant problem highlighted is the abysmal state of logging capabilities in these legacy OT systems. Many devices, particularly older ones, only provide operational data like temperature and pressure readings, completely lacking the granular audit logs necessary for a security investigation. Even when logs exist, they may not contain sufficient information to identify the source device or context, rendering them nearly useless for security analysts. This creates a massive data gap for any potential OT SOC.

The discussion also touches upon the slow pace of modernization. Only a "very small percentage" of OT infrastructure is replaced or modernized annually, meaning the vast majority of systems will remain vulnerable for the foreseeable future. A striking example shared was participating in a red team/blue team exercise on Hook Island, where Remote Terminal Units (RTUs) were found running Windows Server 2012. This outdated operating system, despite its age, was still susceptible to modern attack techniques like PowerShell implants, demonstrating that attackers can leverage familiar IT tools against vulnerable OT targets. The speakers emphasize that organizations must make "mercenary decisions" about where to invest limited time and money, as a complete "rip and replace" of expensive, long-lived industrial assets is often financially unfeasible.

The concept of a dedicated OT SOC emerges from this background as a necessary evolution. While IT environments have established practices for security monitoring, detection, and response, these cannot simply be ported over to OT. The unique characteristics of OT – from device lifespans and logging deficiencies to the critical impact on physical processes and human safety – demand a specialized approach. The problem's existence is further underscored by the low adoption rate of OT SOCs, with only a "fortunate few" in the audience raising their hands when asked if they had one. This gap signifies a widespread vulnerability that organizations are only beginning to address.

Key Findings

▶ Watch: Radics exercise: Windows 2012 RTUs and PowerShell (2:30)

The talk uncovers several critical findings regarding the state and future of OT security and SOC operations:

• Extreme Longevity of OT Assets: Industrial equipment has an average lifespan of 30-40 years, significantly longer than IT assets. This means organizations are managing devices installed decades ago, often lacking modern security features or even basic audit logging capabilities. This legacy burden is a fundamental challenge, as "there's not any capex to rip and replace."
• Pervasive Logging Deficiencies: A major impediment to effective OT security is the absence of adequate security logs. Many devices only output operational data (e.g., temperature, pressure), making it nearly impossible to conduct meaningful security investigations or detect anomalies. The challenge extends to identifying which of a hundred similar devices is sending data, further complicating analysis.
• The IT-OT Hybridization Dilemma: While some advocate for merging IT and OT SOCs, the talk highlights the significant challenges. The risk profiles are fundamentally different: IT typically deals with business risk, while OT involves human lives and physical safety. A unified scoring system might render everything "critical," making prioritization impossible. Furthermore, IT analysts often lack the specialized knowledge of OT systems, leading to missed incidents or incorrect interpretations of alerts in OT-specific SIM tools.
• Importance of OT Context and Relationships: Effective OT incident response relies heavily on understanding the unique operational context. Security analysts, especially those from an IT background, often cannot differentiate between a legitimate operational anomaly (like a "rattlesnake on a power line") and a cyberattack without input from experienced OT operators. Building strong relationships with these "crusty old linemen" is paramount for accurate incident triage and response.
• Lessons from Ukraine's Integrated Security Model: The conflict in Ukraine has forced an accelerated evolution in OT security. The Ukrainians have "cracked the code" by integrating cyber security personnel directly into their bulk power dispatch centers. This physical proximity (a "30-second walk" to talk to an engineer) enables rapid contextualization of alerts and collaborative decision-making, offering a potential new model for hybridized security.
• Asset Management as a Foundational Challenge: Despite advancements, even the "best products" for OT asset management only capture a percentage of devices. A complete and accurate inventory often requires "boots on the ground" and a deep understanding of the physical environment, as diagrams are frequently outdated or non-existent. Without this foundational knowledge, effective security is impossible.
• The Necessity of War Games and Tabletops: The first time an organization deals with a serious OT incident should not be during a real-world "literal fire." Regular tabletop exercises and war games are crucial for preparing incident response teams, including the SOC, to understand critical processes, potential failovers, and the impact of various scenarios. This proactive approach helps avoid situations like the Colonial Pipeline incident, where response capabilities were found wanting.
• Justifying Security as Risk Mitigation: Cybersecurity is typically a cost center, making it difficult to secure funding from executives, particularly CFOs. The speakers advise framing security investments not as an expense, but as risk mitigation. By clearly articulating threats and the severe consequences of inaction (e.g., human lives, prolonged outages), security professionals can compel leadership to acknowledge and fund necessary protections.

Technical Deep Dive

▶ Watch: Practical investment decisions for securing long-life OT (4:00)

The technical discussion in the talk underscores the unique vulnerabilities and complexities inherent in securing Operational Technology (OT) environments. At the core of the problem is the sheer longevity of legacy industrial control systems, many of which were deployed decades ago and continue to operate today. These systems often run on outdated operating systems, such as the Windows Server 2012 found on Remote Terminal Units (RTUs) during a red team exercise. This particular detail highlights a critical vulnerability: while the OS is old, it remains susceptible to modern attack vectors like PowerShell implants, demonstrating that attackers can bridge the gap between contemporary offensive tools and antiquated infrastructure.

A significant technical hurdle is the lack of robust logging capabilities on these legacy devices. Unlike modern IT systems designed with security in mind, many OT components primarily generate operational data—such as temperature, pressure, or flow rates—rather than detailed audit logs essential for security investigations. This means that a crucial source of forensic data, which IT SOCs rely upon heavily, is often entirely absent in OT. Even when some logs are available, they may be generic, lack device-specific identifiers, or be in proprietary formats, making aggregation and analysis challenging for a Security Information and Event Management (SIEM) system. The speakers lament that "you're going to be missing so much data" when trying to monitor these environments.

The conversation also touched upon PKI certificates with 30-to-40-year lifespans used for OT products, further illustrating the long-term design philosophy that clashes with rapid security evolution. This extended validity period can introduce significant challenges for certificate management and revocation in the event of compromise.

The concept of an OT SIM tool is introduced, acknowledging the existence of specialized platforms designed to ingest and analyze data from industrial protocols and devices. However, the technical challenge isn't just about having the tool; it's about the human element. An anecdote reveals an analyst who "didn't know how to use the tool" and a manager who "told them not to use the tool" because they "don't know what to do with it" if an alert is escalated. This highlights a critical gap in skill sets and operational procedures surrounding these specialized tools. The effectiveness of any technology is ultimately limited by the expertise and confidence of its operators.

OT asset management is emphasized as a foundational technical requirement. Knowing "what's in there" is paramount, yet even the "best products" for asset discovery often fall short, only capturing a percentage of the actual devices. This necessitates "boots on the ground" to physically inspect and document the environment, especially given the common absence of accurate engineering drawings. Without a comprehensive and accurate asset inventory, vulnerability management, patch management, and incident scoping become technically impossible.

Finally, the mention of digital twins (e.g., EAPS products) as a potential aid for understanding system behavior and testing scenarios demonstrates an awareness of advanced simulation technologies. While acknowledged as "not perfect," these tools offer a way to model complex OT processes, potentially providing a safer environment for security testing and incident response planning without impacting live critical infrastructure. However, the talk implies that their maturity and widespread adoption in OT security are still developing.

Demo / Proof of Concept

▶ Watch: Accepting 'good enough' for OT log collection (6:10)

The talk did not feature a live demonstration or a specific proof of concept. Instead, the speakers focused on theoretical discussions, real-world anecdotes, and strategic recommendations derived from their extensive experience in OT security and incident response, particularly drawing lessons from global power sector engagements and the conflict in Ukraine.

Defensive Implications

▶ Watch: Why an OT SOC is different from an IT SOC (7:00)

The insights from this talk provide a robust framework for organizations to bolster their defenses in the challenging OT landscape. Defenders must recognize that traditional IT security approaches are often insufficient and require significant adaptation.

1. Strategic Investment Prioritization: Given limited budgets and the inability to "rip and replace" expensive, long-lived OT assets, organizations must make "mercenary decisions." This means identifying the most critical processes and assets and investing security resources where they will yield the "most amount of security for the most return." A thorough risk assessment is paramount to guide these decisions, focusing on potential impact to human safety, environmental damage, and operational continuity.
2. Establish or Evolve the OT SOC: The need for a dedicated or effectively hybridized OT SOC is clear. Organizations without one should prioritize its establishment, recognizing that prevention alone is insufficient. For those with an existing IT SOC, the focus should be on integrating OT-specific expertise and tools. This involves not just acquiring OT SIM tools but critically, training analysts on their use and understanding OT protocols, anomalies, and operational context. Addressing management's comfort level with OT alerts is also crucial to ensure escalations are acted upon.
3. Integrate with Operations (Physical and Procedural): The most impactful defensive strategy highlighted is the deep integration of cybersecurity personnel with operational teams. Inspired by lessons from Ukraine, physically embedding OT SOC analysts within dispatch centers or control rooms, or at minimum fostering strong relationships, is vital. This "30-second walk" enables rapid contextualization of alerts. Security teams should proactively engage with OT operators (e.g., using the "donuts" strategy) to understand normal operating procedures, common anomalies (like the "rattlesnake on a power line"), and the specific quirks of their systems. This relationship building creates a critical feedback loop for accurate incident triage and reduces false positives.
4. Comprehensive OT Asset Management: A foundational defense is knowing what assets are present. Organizations must undertake rigorous OT asset inventory efforts. This goes beyond network-based discovery, often requiring "boots on the ground" to physically identify and document devices, especially for legacy systems where network visibility is poor or non-existent. An accurate asset inventory is essential for vulnerability management, network segmentation, and effective incident scoping.
5. Mandate War Games and Tabletop Exercises: Proactive preparation is non-negotiable. Organizations must regularly conduct tabletop exercises and war games that simulate OT-specific incident scenarios. These exercises should involve not only the OT SOC and IT security teams but also operational personnel, engineering, and leadership. The goal is to test incident response plans, identify communication breakdowns, practice failover procedures, and ensure that all stakeholders understand their roles and responsibilities in a crisis involving critical infrastructure.
6. Justify Security as Risk Mitigation: When seeking budget for OT security initiatives, security professionals must reframe the conversation. Instead of presenting cybersecurity as a cost center, it should be presented as a risk mitigation strategy. By clearly articulating the specific threats, potential consequences (e.g., human casualties, environmental damage, prolonged outages, financial penalties), and the probability of occurrence, CFOs and other executives can be persuaded to invest in reducing unacceptable levels of risk.

Key Takeaways

• OT's Legacy Challenge: Industrial systems have extremely long lifespans (30-40+ years) and often lack modern security features or adequate logging, creating a persistent and complex security challenge.
• OT SOCs are Essential but Rare: A dedicated or hybridized OT SOC is crucial for detection and response in OT environments, yet few organizations currently possess one, highlighting a significant industry gap.
• Context is King: Effective OT security relies heavily on understanding operational context. Security analysts must build strong relationships with OT operators to differentiate between legitimate operational anomalies and cyber threats.
• Integrate Cyber with Operations: The most successful OT security models, as evidenced by Ukraine, physically integrate cyber security personnel with operational dispatch or control centers for rapid information exchange and decision-making.
• Prioritize and Justify: Given limited resources, organizations must make "mercenary decisions" to prioritize security investments based on risk and justify these expenditures to leadership by framing them as essential risk mitigation.
• Practice Makes Perfect: Regular tabletop exercises and war games are indispensable for preparing incident response teams and operational staff for the unique challenges and high stakes of an OT cyber incident.

About the Speaker(s)

The talk featured insights from Adam Robbie and a colleague identified as Joe. While specific titles and companies were not detailed for Joe in the transcript, his contributions revealed a deep expertise in the power sector, having spent "the past 13 years of [his] life working in power grid stuff globally and a lot a lot a lot a lot in Ukraine, especially during the war." This experience provides him with a unique perspective on the operational realities and security challenges faced by critical infrastructure, particularly in high-stress environments. Adam Robbie, as the named speaker, contributed to the discussion on the necessity of OT SOCs and the challenges of integrating security into long-lived industrial systems. Both speakers emphasize the practical aspects of securing OT, drawing from real-world scenarios and the urgent need for effective incident response.