One misrouted network change froze AWS disks for days

transcript

AWS disks are built to heal themselves. In 2011, that healing froze them. For days.

Every EBS disk is copied onto several storage nodes. If a node loses touch with a copy, it assumes it's dead. So it finds free space for a new copy. Finding it usually takes milliseconds.

12:47 a.m. A routine upgrade. Shift traffic off one router. But the traffic goes to the small back-up network. It can't cope. Nodes get cut off. The change is rolled back. And all of them heal. At once.

Here's the catch. Free space runs out in no time. Nodes search again and again. They don't back off enough. A rare bug crashes nodes in the flood. More disks need healing.

About 13 percent of the zone's volumes, stuck. At peak, 45 percent of single-zone databases there. And 0 point 0 7 percent could never be restored consistently.

To heal, the cluster needed room. So AWS moved spare servers in physically. Most volumes were back on 22 April. Disk APIs, on the 23rd.

AWS added spare capacity and made retries back off harder. If everything heals at once, the healing becomes the outage.

Full story in the article.