On 14 August 2003, a blackout hit an area with about 50 million people in eight US states and Ontario. It began in northern Ohio, on the grid of the utility FirstEnergy (FE).
Grid operators learn about trouble from alarms: sounds and on-screen signals when a line trips or a value crosses a limit. Shortly after 14:14 EDT, FE's alarm and logging software failed. FE's own analysis, cited by the US–Canada Task Force, suggested the alarm process had "stalled" while handling an alarm event, as new data piled up and overflowed its input buffers. Its maker, GE Energy, later told The Register's Kevin Poulsen it was a race condition: two processes got write access to the same data structure at once, and the alarm program went into an infinite loop.
Nothing announced that the alarms had stopped. At 14:41 the server running them failed, and the alarm program moved to the backup server "still stalled and ineffective", so the backup failed 13 minutes later, at 14:54. IT staff were paged but didn't tell the control room. A warm reboot at 15:08 looked fine, yet the alarms stayed frozen. Only at 15:42 did a control room operator tell IT about the alarm problem.
Meanwhile, from 15:05, FE's high-voltage lines touched overgrown trees and tripped, one after another. With no alarms, operators trusted their outdated picture and even discounted calls from neighbours about line trips. At 16:05:57 the loss of the Sammis-Star line set off the cascade: within seven minutes it rippled across much of the northeast US and Canada.
- An area with about 50 million people and 61,800 MW of load affected
- 14:14 EDT: the alarms stopped, with no warning
- 13 minutes: how long the backup server lasted
- By 16:13, more than 508 generating units at 265 power plants lost
- $4 billion to $10 billion estimated cost in the US
Why it matters: Silence from a monitoring system isn't good news. Alert when the alerts stop, and remember that failing over doesn't help if the backup inherits the same broken state.