Four packets that never expired took down CenturyLink, and 911, for 37 hours

transcript

Four malformed packets cut off reliable 911 for millions of Americans. They never expired. So they never stopped.

27 December 2018. A module in Denver creates four malformed packets. Nobody knows why. Each one is addressed to every connected node. Valid header. Valid checksum. Over 64 bytes, so the filter lets them through. And no expiration time. Nothing will ever drop them.

They travel on a management channel CenturyLink never used. It was on by default. Every node forwards them to all its neighbours. Even back to Denver. They circle forever. And multiply. The nodes drown. They stop routing traffic. Engineers can't even log in.

3:56 a.m. A customer near New Orleans reports a problem. Engineers go to Omaha and Kansas City to log in directly. 9:02 p.m. They find and remove the Denver module. But here's the twist. It doesn't end. The packets keep echoing. After midnight, nodes are told to ignore them. The channel is switched off.

Up to 22 million customers in 39 states. About 17 million without reliable access to 911. At least 886 emergency calls never got through.

Anything that forwards messages needs a hop limit or an expiry. And turn off features you don't use. They still run.

Full story in the article.