Four packets that never expired took down CenturyLink's network, and 911 with it, for 37 hours.

On 27 December 2018, CenturyLink's fiber network failed nationwide for almost 37 hours. Up to 22 million customers in 39 states were affected, about 17 million of them without reliable access to 911. At least 886 calls to 911 were not delivered.

The network's nodes, made by Infinera, had a proprietary management channel for very fast automatic rerouting. It came enabled by default. CenturyLink knew about it but never configured or used it. A filter kept packets of 64 bytes or fewer, the size of normal management packets, off the channel.

Early that morning, a switching module in Denver spontaneously generated four malformed management packets. Neither CenturyLink nor Infinera knows how or why. Each one had a broadcast address (deliver to every connected node), a valid header and checksum, no expiration time, and a size over 64 bytes. So each passed every check.

Every node that received them sent them on to all its neighbours, including back to Denver. With no expiry, they never died; they multiplied. The flood ate the nodes' processing power until they lost internal synchronisation and stopped routing traffic. And because the nodes were overloaded, engineers couldn't log in to them remotely.

CenturyLink learned of the problem from a customer near New Orleans at 3:56 a.m. It sent engineers to Omaha and Kansas City to log in to nodes directly, traced the packets to Denver, and removed the faulty module at 9:02 p.m. That didn't end it: the packets were still echoing between nodes. Recovery came after 12:09 a.m. on 28 December, when engineers told nodes to ignore the packets and disabled the management channel.

Why it matters: Anything that forwards messages needs a hop limit or an expiry, or one bad message lives forever. Turn off features you don't use, and keep a way into your network that doesn't depend on the network.