Amazon S3 is AWS's service for storing files, which it calls objects. On 28 February 2017, its team in the Northern Virginia region (US-EAST-1) was chasing a small problem: the S3 billing system was running slower than expected.
At 9:37 AM (US Pacific time), an authorized engineer, following an established playbook, ran a command to remove a small number of servers from one of the subsystems the billing process uses. In AWS's words, "one of the inputs to the command was entered incorrectly", and a much larger set of servers was removed.
Those servers belonged to two other subsystems that everything depends on. The index subsystem knows where every object in the region lives; without it S3 can't read, list, write or delete anything. The placement subsystem decides where new data goes. With so much capacity gone, both needed a full restart, and while they restarted, S3 in the region could not serve requests.
The restart was slow for a simple reason: AWS had not fully restarted these subsystems in its larger regions "for many years", and S3 had grown enormously since. Other AWS services in the region that store data in S3 were hit too, including new EC2 server launches and Lambda. Even AWS's own status page couldn't be updated until 11:37 AM, because it depended on S3.
Reads came back fully at 1:18 PM, and writes at 1:54 PM.
AWS changed the tool to remove capacity more slowly and to refuse any removal that would take a subsystem below its minimum safe size. It also brought forward work to split the index into smaller pieces, which it calls cells, so that one failure touches less and restarts faster.
- 9:37 AM: the command runs
- 1:18 PM: reads, lists and deletes fully back
- 1:54 PM: writes back, about 4 hours 17 minutes later
Why it matters: People will mistype. The fix is not "be more careful" but tools that refuse dangerous inputs, recovery paths you actually rehearse, and smaller pieces (cells) so one mistake has a small blast radius.