The scariest failure on Prime Day isn't a disk. It's us.

transcript

On Prime Day, the scariest failure isn't a disk. It's a bad deploy. And every replica runs the same bug.

A new checkout build crashes on one rare cart. It crashes a server. The retry crashes the next. The survivors fall too. Every customer is down.

Now say 40 cells. Each a full copy of checkout. The bad cart reaches only its cell. 2.5 percent, at worst. And deploys go one cell at a time.

AWS asks: every customer at a 5 percent failure rate? Or 5 percent of customers at 100 percent? Cells pick this.

But here's the catch. Every cell shares one price service. So each cell gets its own 4 of its 32 groups. 35,960 ways to pick them. No two cells share more than 2.

A poison request kills its cell's 4 groups. Every other cell still has at least 2. One cell hurt.

Replication copies your mistakes. Cells turn an outage for everyone into one in forty.

Full breakdown in the lecture.