One switch came back online and jammed AT&T's network for nine hours

transcript

One phone switch fixed itself. That's what broke A T and T. Its comeback jammed A T and T's long-distance network. For about nine hours.

2:30 p.m. A New York switch hits a minor hardware fault. It pauses new calls for four to six seconds. Routine. Then it's back. It starts sending calls again. And its neighbours start falling over.

Its first call tells a neighbour: New York is back. The neighbour updates its status. For a few seconds, it's vulnerable. Two calls within a hundredth of a second? Data damaged. It restarts. When it comes back, it does the same to its neighbours. Again and again. Across the whole network.

A T and T said every 4ESS switch had the flaw. All 114. About half of their long-distance callers got a busy signal.

Here's the twist. The flaw came in a mid-December update. Its job? Reach the backup signalling network faster. It was tested in the lab. But this combination of events had never happened. Until that day. And traffic wasn't even heavy. Lower than a normal Monday.

Engineers cut the flood. They suspended signalling on the backup links. 11:30 p.m. The last link cleared. Next day, the old program went back.

Teams test what happens when a node fails. Test what happens when it comes back. Same code everywhere spreads it everywhere.

Full story in the article.