The 2003 blackout grew in a control room whose alarms had silently stopped

transcript

A blackout across an area of 50 million people began with silence. The control room alarms stopped. And nothing said so.

Ohio, 14 August 2003. FirstEnergy runs the local grid. 14:14. The alarm software freezes. No beep. No warning. 14:41. The server running it fails. It moves to the backup. 13 minutes later, the backup fails too. Only at 15:42 does an operator tell IT about the alarms.

GE later named the villain. Two processes writing the same data at once. A race condition. The alarm program spun in an infinite loop. New data piled up and overflowed its buffers. Here's the twist. Failover moved the program to the backup. Still stuck.

On their screens? No alarms. Everything looked calm. From 15:05, high-voltage lines touched trees and tripped. One after another. Neighbours called about the trips. The operators trusted their screens.

16:05:57. One more line trips. The cascade begins. Within seven minutes, it ripples across the northeast US and Canada. Over 508 generating units at 265 power plants. Lost.

Silence from your monitoring is not good news. Alert when the alerts stop. And check what your backup inherits.

Full story in the article.