Netflix kept streaming through the April 2011 AWS outage that hampered popular sites.

On Thursday 21 April 2011, Amazon Web Services had a large outage in its US-East region. It took down or severely hampered a number of popular websites. Netflix, which ran its streaming systems on AWS, kept going: the service "ran without intervention", Netflix wrote, with higher error rates and latency than usual through the morning, its quiet time, and no noticeable rise in customer service calls.

Netflix had avoided EBS, Amazon's network disks, as its main data store, and the services it relied on, SimpleDB, S3 and Cassandra, were not affected. The rest was design:

• Stateless services: any instance can serve any request, so a lost server is simply replaced.
• Data across zones: where state was unavoidable, redundant hot copies sat in different zones.
• Graceful degradation: aggressive timeouts so failing parts fail fast; fallbacks, such as cached or non-personalised rows of movies; and dropping non-critical features that turn slow.
• N+1 capacity: systems spread across three of the four zones, all active, with more capacity reserved than used so the others could absorb the loss of a whole zone.

It wasn't painless. When it became clear AWS wouldn't fix things before Netflix's early-evening peak, engineers decided to move traffic out of the failing zone by hand. Their tools weren't built for wholesale changes, so every service team made manual, "potentially error prone" changes, including updating all of the load balancer endpoints.

So Netflix's lessons were to automate zone failover into a "one click" operation, and to create more failures. It already ran Chaos Monkey, a service that kills other services so engineers get used to failure. But nothing simulated losing a whole zone. People were already calling that idea "Chaos Gorilla".

Why it matters: Design for the failure, then practise it. A lost zone was part of Netflix's plan, and it still found a gap: the failover worked, but only by hand.