Virtual machines on Windows Azure ran a small program, the guest agent, that talks to a host agent on the physical server. When a new VM started, the guest agent's first job was to create a security certificate valid for one year.
It worked out "one year from today" by taking today's date and adding one to the year. On 29 February 2012, that gave 29 February 2013, a date that doesn't exist. Creating the certificate failed, and the guest agent quit.
Then the system's own self-healing spread it. The host agent waited 25 minutes to hear from the guest agent, then reset the VM and tried again. After three failures in a row, it decided the hardware was broken and flagged the server for "Human Investigate". The cluster controller then moved that server's VMs to healthy servers, where they hit the same bug and flagged those servers too.
The bug fired at 4:00 PM Pacific time on 28 February, midnight UTC on the 29th. Clusters in the middle of rolling out new agent versions hit it straight away, and 75 minutes later (three times 25 minutes) alarms stopped the updates and alerted engineers. At 6:55 PM, for the first time ever, Microsoft switched off customers' ability to deploy or change applications worldwide.
The fix was ready overnight, but the rush caused a second outage. In seven partly updated clusters, engineers pushed a repair to every server at once; it contained an incompatible networking plugin, and every VM there, healthy ones included, lost its network. All services were healthy again by 2:15 AM on 1 March. Microsoft gave every customer of the affected services a 33% credit for the month.
- 25-minute timeout x 3 tries = 75 minutes until alarms fired
- 7 clusters knocked offline by the rushed fix
- 33% credit to all customers of the affected services
Why it matters: Don't hand-roll date math. And self-healing can spread a software bug, so it needs a brake that stops and calls a human when every "fix" fails the same way.