AT&T's long-distance network spent nine hours knocking itself over.

At about 2:30 p.m. EST on 15 January 1990, one of AT&T's 4ESS long-distance switches in New York City had a minor hardware problem. Its normal recovery paused new calls for four to six seconds, and it asked connected switches not to send it calls meanwhile. Routine.

When ready, it simply started sending calls again. For a neighbouring switch, the first call from New York meant "New York is back", so it began updating its status map. A software flaw left the processor doing that update vulnerable for several seconds. If two call messages from New York arrived within 1/100th of a second in that window, data got damaged and the processor was taken out of service to restart.

Its duplicate took over and was hit the same way, cutting that switch off from the signalling network. When that switch recovered, it sent call attempts to its neighbours and set off the same failure in them. AT&T said the effect "cascaded through the network", fed by ordinary calls. Calling volume was lower than a normal Monday's.

The flaw had reached every 4ESS switch, all 114, according to AT&T's Larry Seese, in a mid-December update. It was meant to make switches reach a backup signalling network faster, and was tested in the lab, but this combination of events had never occurred.

Engineers stabilised the network by suspending signalling on the backup links, cutting the flood of messages. The last link was cleared at 11:30 p.m. The next day they put the previous program back. AT&T said about half of long-distance callers on its network got a busy signal or an "all circuits are busy" recording.

Why it matters: Test what happens when a failed node comes back, not just when it fails. When every node runs the same code, one node's comeback can knock over all the rest.