One wrong input took Amazon S3 down for hours

transcript

One wrong input. And Amazon S3 stopped answering. For hours. The engineer was authorized. Following the playbook. It still happened.

28 February 2017. S3's billing system in Virginia is running slow. 9:37 AM. An engineer runs a command. Remove a few servers. But one input is wrong. A much larger set disappears. And S3 in the whole region stops serving requests.

Those servers ran the index. It knows where every object lives. Without it, S3 can't read, list, write or delete. Anything. Placement went too. That decides where new data goes. Then everything built on S3 falls. New EC2 launches. Lambda.

Both subsystems need a full restart. But here's the catch. AWS hadn't fully restarted them in big regions for many years. And S3 had grown enormously since. So the restart crawled.

Even AWS's status page was stuck until 11:37. It depended on S3. Reads came back fully at 1:18 PM. Writes at 1:54. About 4 hours 17 minutes after one command.

So what did AWS fix? Not the engineer. The tool. It now removes capacity more slowly. It refuses any removal below a subsystem's minimum safe size. And the index is split into small pieces, called cells. One failure touches less.

People will mistype. Be more careful is not a fix. Build tools that refuse. Rehearse recovery. Keep the pieces small.

Full breakdown in the lecture.