Netflix kept streaming through the April 2011 AWS outage that hampered popular sites

transcript

April 2011. Amazon's cloud has a large outage. Popular websites go down. Netflix, running on that same cloud, keeps streaming.

21 April. AWS US-East. Many sites go down or limp along. Netflix ran without intervention. More errors than usual, but it ran. No noticeable rise in customer service calls. How?

Netflix spread its systems across three of four zones. All active. And it reserved more capacity than it actually used. So if one whole zone fails, the others absorb it.

Stateless services. Any server can answer any request. Lose one, replace it. Aggressive timeouts. Can't personalise your rows? Show cached ones. And a slow, non-critical feature just gets dropped.

But here's the catch. It wasn't painless inside. AWS wasn't going to fix it before Netflix's evening peak. So every service team moved traffic out of the zone. Manually. It worked. But manual changes are potentially error prone.

Netflix already ran Chaos Monkey. A service that kills other services. But nothing practised losing a whole zone. They called that idea Chaos Gorilla.

Netflix planned to make zone failover one click. Design for the failure. Then practise it, until nobody has to do it by hand.

Full story in the article.