Amazon EBS gives cloud servers their disks. Each disk ("volume") is copied onto several storage nodes. When a node loses touch with a copy, it assumes it's dead and finds another node to make a new copy on; finding one normally takes milliseconds. AWS calls that re-mirroring. While it happens, the volume is blocked, so it looks "stuck".
The nodes are linked by two networks: a fast primary one and a lower-capacity back-up one, never meant to carry everything.
At 12:47 a.m. PDT on 21 April 2011, during a routine upgrade in one Availability Zone of US East, engineers needed to shift traffic off one router. The shift "was executed incorrectly": instead of the other primary router, traffic went onto the low-capacity back-up network, which couldn't take it. Many storage nodes were cut off from each other completely.
When the change was rolled back, all those nodes started re-mirroring at once. The cluster's free space ran out fast, and nodes got stuck in a loop, searching again and again for space that wasn't there, without backing off enough. A rare race condition then crashed nodes under the flood, creating even more volumes to re-mirror. About 13% of the volumes in that zone were stuck. The storm also starved the regional control plane of threads, so for periods disk APIs saw high error rates across the whole region.
By midday the cluster was stable, but healing it meant physically moving spare servers in from across the region. Most volumes were back by 12:30 p.m. on 22 April; disk APIs in the zone returned at 6:15 p.m. on 23 April. In the end 0.07% of the zone's volumes could not be restored to a consistent state. At peak, 45% of single-zone RDS databases there were stuck too.
AWS's fixes: far more spare capacity, harder back-off, and a fix for the race condition.
- About 13% of volumes in the zone stuck
- About 2.2% still not restored by 12:30 p.m. on 22 April
- 0.07% of the zone's volumes could not be restored consistently
- 45% of single-zone RDS databases in the zone hit at peak
Why it matters: Self-healing needs room to heal. If every replica recovers at once, recovery itself becomes the outage, so keep spare capacity and make retries back off.