One command on one router shook Microsoft's whole global network

transcript

The engineer could touch exactly one router. One command later, every router in Microsoft's WAN was recomputing.

Microsoft's network had safety nets. A four-step change procedure. Access to only one device at a time. And every command approved live, against a block list.

But the procedure had been changed. And not re-tested. 07:08. An engineer adds capacity in Madrid. Runs a command to purge routing data. Alerts fire within 5 minutes. Nobody tells the engineer. 33 minutes later, same step on the second router. A second wave.

Here's the twist. On two vendors' routers, that command stays local. On the third vendor's, it runs on every router in the network. Nobody had caught that. So it wasn't blocked.

One router. But it's connected to all the rest. Routers worldwide start recomputing the network. Then internet routes, too. Azure and Microsoft 365 users get timeouts and lost connections.

08:20. They find the command. 09:05. Most regions are back. But health systems had paused. Packet loss lingers until 12:43.

One device at a time limits who you touch. Not what your command touches. Know what every command does. On every platform.

Full story in the article.