Fear vs Physics: Diagnosing Grid Chaos

Fear vs Physics: Diagnosing Grid Chaos

Speakers: Emma Stewart, Chief Grid Scientist, Idaho National Lab

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=930l_omgN1w

Overview

Emma Stewart, Chief Grid Scientist at Idaho National Lab, delivered a compelling talk titled "Fear vs Physics: Diagnosing Grid Chaos" at DEF CON, challenging the prevalent tendency within the cybersecurity community to immediately attribute major power grid outages to cyberattacks. With two decades of experience in power systems across multiple countries, including extensive work in incident response for the electric grid, Stewart expressed profound frustration with the pervasive Fear, Uncertainty, and Doubt (FUD) that often overshadows the actual, physics-based causes of grid disturbances. Her presentation served as a critical educational intervention, aiming to re-center the conversation around the fundamental engineering principles that govern grid stability and failure.

The core of Stewart's argument is that while cyber threats to critical infrastructure are real and warrant attention, a deep understanding of power system physics is paramount for accurate incident diagnosis and effective defensive strategies. She meticulously deconstructed several high-profile outages, most notably the massive blackout that swept across Spain and Portugal in April 2022, demonstrating how these events were driven by well-understood physical phenomena rather than malicious cyber intrusion. This talk is crucial for cybersecurity professionals, policymakers, and the public alike, as it underscores the importance of analytical rigor over speculative attribution in safeguarding one of society's most vital infrastructures.

Background

▶ Watch: Introduction and motivation for 'Fear vs Physics' talk (0:00)

The United States electric grid is a colossal and intricately interconnected system, comprising approximately 3,000 utilities, 26,000 generators, and half a million miles of transmission lines. Despite its complexity, Stewart asserts that the U.S. grid remains one of the most reliable in the world, a testament to decades of engineering and operational expertise. However, this system is in constant evolution, transitioning from large spinning machines to solid-state devices and embracing distributed controls and advanced communications. This modernization brings new efficiencies but also introduces new complexities that must be understood in context.

Despite the grid's general reliability, outages do occur, and their causes are often far more mundane and rooted in physics than popular narratives suggest. Stewart highlighted numerous examples: a woman climbing a transformer in the past year caused an outage for 800 homes; copper theft, squirrels (dubbed "super squirrels" due to their persistent impact), and even a bird dropping a fish on a power line have all been documented causes of localized power loss. Weather events, equipment aging, and fires are also frequent culprits, visibly knocking down distribution lines or causing flammable transformers to ignite.

Against this backdrop of physics-driven incidents, a concerning trend has emerged: the immediate and often unsubstantiated attribution of major outages to cyberattacks. Stewart cited the example of a recent Texas outage where some speculated about a "surprise sunset," illustrating the bizarre misinformation that can circulate. More critically, she pointed to the April 2022 blackout in the Iberian Peninsula, where, within minutes of the widespread power loss, social media and even well-known cybersecurity professionals were quick to declare it a Russian cyberattack. This knee-jerk reaction, often accompanied by product sales pitches, not only spreads public panic but also diverts attention and resources from the actual root causes, which are frequently identifiable through established power system physics. Stewart's talk was born from her profound frustration with this pattern, emphasizing the need for a fact-based, physics-informed approach to incident analysis.

Key Findings

▶ Watch: Overview of US grid reliability and common outage misinformation (1:30)

Stewart's key findings unequivocally underscore that the vast majority of grid outages, including those that garner significant media attention, are driven by fundamental physics and engineering challenges rather than cyberattacks. This perspective is crucial for accurately diagnosing incidents and developing effective defensive strategies.

The central case study presented was the Iberian Peninsula blackout on April 28, 2022, which saw Spain and Portugal plunged into darkness. Stewart meticulously detailed the sequence of events, demonstrating a clear physics-based progression:

• Forecast Mistake and Improper Dispatch: Early warning signs emerged between 9:00 a.m. and noon, indicating a fundamental miscalculation in the day's generator dispatch—how much power generators were expected to produce. This led to an imbalance in the system.
• Abnormal Voltage Fluctuations: As a direct consequence of the dispatch error, the system experienced abnormal voltage fluctuations, with voltage "bouncing up and around." This instability caused numerous distributed solar installations on homes to trip offline, not due to a cyberattack, but as an intended protective response to protect the equipment from damage due to off-nominal voltage.
• System Oscillations: Between 12:00 p.m. and 12:30 p.m., the grid began to exhibit oscillations—a periodic "wiggling" of power. These oscillations were detected by neighboring France, which, in an attempt to protect its own grid and potentially provide damping to Spain, disconnected a specific tie line. However, because Spain's generators were improperly dispatched, this action failed to dampen the oscillations, instead exacerbating the instability.
• Cascading Collapse: Around 12:32 p.m., the system rapidly collapsed. Critical infrastructure, particularly nuclear generators, performed an emergency shutdown known as a scram to prevent catastrophic damage. This protective action triggered a cascading failure of protection systems across the entire country, leading to a full blackout within a mere two minutes.
• Successful Black Start: Despite the dramatic collapse, the post-event response was a success. Utilities initiated a black start procedure, using specialized generators that can start without external power. They systematically rebuilt the network, and power was fully restored across Spain by 4:00 a.m. on April 29th—a remarkable 16-hour restoration period, significantly faster than some comparable U.S. events.

Stewart highlighted that the "lights going out" is often an intended defense mechanism in power systems, designed to protect expensive equipment from severe damage during instability. The true worst-case scenario, she argued, is when the lights stay on and equipment burns up.

Crucially, Stewart identified misinformation and FUD as significant threats to effective grid security. She cited numerous instances where unfounded cyberattack claims caused panic and misdirected attention:

• Reports of "Rogue hardware and Chinese inverters" containing a "kill switch" (which was merely a National Electric Code-mandated rapid shutdown device).
• The Oldsmar water system incident, initially widely reported as a cyberattack poisoning the water, later confirmed to be human error.
• A massive transformer failure in London, attributed to Russian or Chinese cyberattacks, but actually caused by excessive moisture.
• Heathrow Airport's power loss, a planning failure due to inadequate switching capabilities to backup substations.
• A surge of "vulnerability to massive power grid outage to buy my product" news stories coinciding with security conferences, creating an environment of fear that distracts from genuine threats.

Stewart's findings serve as a stark reminder that while cyber threats are a component of the modern risk landscape, understanding the underlying physics of grid operation is fundamental to distinguishing real threats from sensationalized claims and ensuring appropriate responses.

Technical Deep Dive

▶ Watch: Common physical causes of power outages, not cyberattacks (2:50)

The fundamental principle governing the stability of an electric power grid is the continuous and precise matching of generation and load. Any imbalance, even slight, can lead to deviations in voltage and frequency, which are critical parameters for stable operation. When these parameters drift too far from their nominal values, the system becomes unstable.

Stewart delved into the specifics of oscillations, which were a direct precursor to the Spanish blackout. Oscillations are essentially periodic fluctuations or "wiggling" of power and voltage across the system. She used the relatable analogy of driving a car: if a car hits a large pothole, the balance of its tires can be knocked off, causing the steering wheel to shake. In the grid, this shaking indicates instability. Oscillations are typically triggered by a significant system change, such as the sudden loss of a large generator or a major transmission line. If these oscillations are not adequately damped—meaning their amplitude is not reduced over time—they can grow out of control, leading to a cascade of protective actions and system collapse.

The root causes of oscillations are multifaceted, often involving:

• Rotor angles interfering with each other: Generators on the grid rotate in synchronicity. When this synchronicity is disturbed, their rotor angles can begin to oscillate relative to each other, causing power flows to fluctuate.
• Weak ties between areas: Interconnections between different parts of the grid, if not robust enough, can become points of instability where oscillations can easily propagate and amplify.
• Other forces and weather: Environmental factors, particularly temperature variations, can significantly impact the electrical characteristics of transmission lines and equipment, contributing to system instability and the onset of oscillations. Stewart specifically debunked the idea that a single, isolated cyber event could cause a widespread oscillation, emphasizing that it is typically a complex interplay of these factors.

The talk also clarified the phenomenon of solar trips. During the Spanish event, many residential solar installations tripped offline, leading some to speculate about a cyberattack. Stewart explained that this is a normal and intended function. When the grid experiences abnormal voltage fluctuations—either too high or too low—solar inverters are designed to disconnect from the grid to protect themselves and the overall system. This is a safety and equipment protection feature, not an indicator of malicious activity. The warning messages seen by homeowners, such as "high voltage warning and trip," are precisely what these devices are engineered to do.

Protection systems are the grid's last line of defense. These automated systems continuously monitor electrical parameters and are programmed to isolate faulty sections or components to prevent widespread damage. In the Spanish blackout, when the oscillations became uncontrollable, nuclear generators initiated a scram—an emergency shutdown designed to quickly and safely halt the nuclear reaction and prevent damage to the plant. This, in turn, led to other protection systems across the country operating sequentially, ultimately resulting in the full blackout. The purpose of these systems is not to keep the lights on at all costs, but to protect the integrity of the generators and bulk system, ensuring that a black start and subsequent restoration are possible.

The black start procedure itself is a critical engineering feat. After a complete blackout, the system cannot simply be switched back on. It requires a methodical process:

1. Assessment: Operators first assess the extent of the damage and identify what equipment is still operational.
2. Black Start Plants: Specialized generators, known as black start plants, which do not require an external power source to start, are brought online. These often include hydro, diesel, or smaller gas turbines.
3. Network Reconstruction: These black start units then energize small sections of the grid, gradually building up the network piece by piece. This involves sequentially bringing transmission lines, substations, and larger generators back online until the entire system is reconnected and stable. The 16-hour restoration in Spain was highlighted as an impressive demonstration of this complex and well-executed engineering process.

Stewart's technical deep dive underscored that understanding these intricate physical and engineering principles is not merely academic but absolutely vital for anyone involved in protecting the grid. Without this understanding, the cybersecurity community risks misinterpreting events, chasing phantom threats, and ultimately failing to address the real vulnerabilities that exist.

Demo / Proof of Concept

▶ Watch: Iberian Peninsula blackout: Immediate cyberattack speculation (4:10)

No live demonstration or proof of concept was presented during Emma Stewart's talk. The presentation's focus was entirely on the analytical deconstruction of real-world power grid events and the explanation of underlying physics principles, rather than showcasing specific tools or attack vectors. Stewart explicitly noted the absence of slides during her presentation, further emphasizing the reliance on verbal explanation and conceptual understanding over visual or interactive demonstrations.

Defensive Implications

▶ Watch: Iberian blackout: The real physical chain of events (6:00)

The defensive implications derived from Emma Stewart's talk are profound and call for a significant reorientation of focus within the cybersecurity and critical infrastructure communities. The primary takeaway is the absolute necessity of grounding cybersecurity efforts in a deep understanding of power system physics. Without this foundational knowledge, defenders risk misinterpreting incidents, misallocating resources, and falling victim to sensationalized narratives.

First and foremost, combatting misinformation is a critical defensive strategy. The immediate attribution of outages to cyberattacks, often fueled by social media and ill-informed experts, causes widespread public panic and distracts from the actual work of diagnosis and restoration. Stewart highlighted the stark contrast between the panic in Spain, where people drained stores believing they'd be without power for days, and a masterclass in communication demonstrated by a utility in South Carolina. After a major transformer was shot, cutting power to 47,000 customers, the CEO provided clear, continuous updates on the situation, the engineering efforts (building 2 miles of new line in 12 hours), and realistic timelines. This transparent communication fostered community support, with residents feeding line workers, rather than panic or hostility. Defenders must adopt similar proactive and accurate communication strategies during incidents.

Secondly, Stewart urged a shift towards prioritizing real threats over speculative ones. She criticized the constant barrage of news stories about "rogue hardware," "Chinese inverters," or "kill switches" (which are simply mandatory rapid shutdown devices). These narratives, often coinciding with major security conferences, create FUD that overshadows genuine, technically validated vulnerabilities. She specifically pointed to "really interesting work on relays" by Mandy and Chris, suggesting these are the types of credible threats that asset owners should be focusing on, rather than being "so busy running after the fear, uncertainty, and doubt." This implies a need for cybersecurity research and intelligence to be more closely aligned with operational realities and engineering principles, delivering actionable insights rather than alarmist headlines.

Thirdly, incident response protocols must evolve to ensure objective root cause analysis. Stewart stressed that incident investigations should never start with an assumed trigger, such as "it was definitely a cyber event." Instead, the process must begin with a comprehensive effort to find the true root cause, whether it's a weather event, equipment failure, human error, or indeed, a cyberattack. This requires interdisciplinary teams that include power engineers and cybersecurity experts working collaboratively, each bringing their specialized knowledge to the table.

Finally, the talk emphasized the importance of resilience and planning. The Spanish blackout, while physics-driven, also exposed "pretty big resilience failures," including poor communication and the loss of communication towers. This highlights the need for robust and redundant communication infrastructure. Moreover, planning failures, such as Heathrow Airport's inability to switch to backup substations during an outage, underscore the importance of comprehensive contingency planning and regular drills for various scenarios, including full system black start procedures. The 16-hour black start in Spain was presented as a success story in this regard, demonstrating the value of well-trained personnel and established protocols.

In essence, defensive strategies for the power grid must move beyond a narrow cybersecurity lens to embrace a holistic understanding of the complex interplay between cyber, physical, and human factors. This means fostering greater collaboration between power engineers and cybersecurity professionals, investing in physics-informed threat intelligence, prioritizing transparent communication, and rigorously focusing on real-world vulnerabilities and robust resilience planning.

Key Takeaways

• Physics, Not Cyber, Dominates Outages: The vast majority of power grid outages, including major ones like the Iberian Peninsula blackout, are attributable to fundamental power system physics, engineering failures, and environmental factors, not cyberattacks.
• Misinformation is a Major Threat: Premature and unsubstantiated attribution of outages to cyberattacks creates widespread public panic, erodes trust, and diverts critical resources and attention from actual, physics-based root causes and real cyber threats.
• Grid Protection is Intentional: Automatic shutdowns and trips, such as nuclear plant scrams or solar inverter disconnections due to voltage fluctuations, are often intended defense mechanisms designed to protect expensive equipment and enable successful system restoration, not indicators of catastrophic failure.
• Interdisciplinary Understanding is Crucial: Cybersecurity professionals working in critical infrastructure must develop a foundational understanding of power system engineering principles, including concepts like generation-load balance, voltage/frequency control, oscillations, and protection systems, to accurately diagnose and respond to incidents.
• Transparent Communication is Key to Resilience: Effective, clear, and honest communication from utilities during an outage can prevent mass panic, build community support, and facilitate the restoration process, as exemplified by the South Carolina utility's response.
• Prioritize Real, Validated Threats: Focus defensive efforts on documented and technically sound vulnerabilities (e.g., in relays) rather than being swayed by sensationalized narratives, exaggerated claims, or Fear, Uncertainty, and Doubt (FUD) surrounding speculative threats like "kill switches" or "rogue hardware."

About the Speaker(s)

Emma Stewart is the Chief Grid Scientist at Idaho National Lab, bringing over 20 years of extensive experience in power systems across multiple countries. She holds a PhD in electrical engineering and a Masters in electromechanical engineering, demonstrating a deep academic and practical understanding of complex grid operations. Throughout her career, Stewart has been actively involved in incident response for the electric grid, witnessing firsthand the challenges and misconceptions surrounding grid failures. Her talk, "Fear vs Physics: Diagnosing Grid Chaos," was born out of her frustration with the cybersecurity community's tendency to prematurely attribute power outages to cyberattacks, often overlooking the fundamental physics that govern these critical infrastructures. Her expertise lies in bridging the gap between power engineering and cybersecurity, advocating for a physics-informed approach to grid security.