Adding 1 to the year broke Windows Azure on leap day

transcript

To get one year from today, just add 1 to the year. Easy, right? On 29 February 2012, that line broke Windows Azure.

Every new Azure VM ran a guest agent. First job: a certificate. Valid one year. So the code took today, and added 1 to the year. Today was 29 February 2012. So: 29 February 2013. That date doesn't exist. The certificate failed. The agent quit.

One broken VM. Annoying, but small. Right? But here's the catch. Azure's own self-healing spread the bug.

The host agent waits 25 minutes. Silence. So it resets the VM. Three failures in a row? It decides the hardware is broken. The server is flagged. Its VMs move to healthy servers. Where they hit the same bug. And flag those servers too.

4 PM Pacific on 28 February. Midnight UTC. The bug fires. Alarms fire 75 minutes later. Three tries, times 25 minutes. 6:55 PM. Microsoft switches off deploys worldwide. A first, ever. Then a rushed fix cuts the network in seven clusters. Healthy VMs too.

Everything was healthy by 2:15 AM. Affected customers got a 33 percent credit. Lesson one: don't hand-roll date math. Lesson two: when every fix fails the same way, stop. Call a human.

Full breakdown in the lecture.