Microsoft's global Wide Area Network (WAN) carries traffic between its regions and to the internet. Its routers share a map of the network through a routing protocol called IGP, and changes follow a four-step procedure: emulator test, lab test, written review, then deploy with access to only one device at a time. On top of that, every command needs real-time approval, with a block list of commands that have global impact.
On 25 January 2023 at 07:08 UTC, an engineer was adding capacity in Madrid. The procedure had been changed before the work, and Microsoft says the changed version "was not re-tested" and lacked proper checks. It now included a command to purge the IGP database. On routers from two manufacturers, that command affects only the local router. On the third manufacturer's routers, it runs across all routers joined by IGP, ordering every one to recompute its view of the network. Nobody had caught that difference, so it wasn't on the block list.
The engineer had access to just one router, but that router was connected to all the rest. Routers worldwide started recomputing, then internet routes had to be re-learned too. Monitoring raised alerts within 5 minutes, but the engineer wasn't told, and ran the same step on the second Madrid router 33 minutes later: a second wave.
Customers saw timeouts, long delays or lost connections to Azure, Microsoft 365 and Power Platform. The problem command was found by 08:20; most regions had recovered by 09:05. But automated systems that keep the WAN healthy had paused during the chaos, and until they were restarted by hand some paths lost packets. Everything was mitigated at 12:43 UTC.
Microsoft's fixes: block similar commands for all three vendors, and show on-call engineers live who is changing what.
- 07:08 to 12:43 UTC of connectivity problems; most regions back by 09:05
- Alerts within 5 minutes; second router changed 33 minutes after the first
- About 1 hour 40 minutes for the network to restore connectivity to every prefix
- 1 of 3 router vendors ran the command network-wide
Why it matters: "One device at a time" limits who you touch, not what your command touches. Know what every command does on every platform, and tell the person making the change the moment alarms go off.