Google takes its Chubby lock service down on purpose when it has been too reliable

transcript

Google had a service so reliable, it became a problem. So now they take it down. On purpose.

Meet Chubby. Google's lock service. Systems use it to coordinate. The global one keeps each replica in a different region. It almost never fails. So teams keep adding dependencies. Assuming it would never go down.

But here's the catch. Rarely isn't never. When global Chubby did fail, services broke. Users saw outages.

The obvious fix? Make Chubby even more reliable. Google did the opposite. Meet the availability target. But don't significantly exceed it.

So here's the rule. Every quarter, they check. Has a real failure dropped availability below the target? If not, SRE takes Chubby down. A controlled outage.

It flushes out bad dependencies soon after they're added. Services that can't cope find out on a planned day. And owners face the reality of distributed systems. Sooner, not later.

People build on how your service really behaves. Not what you promise. If it never fails, nobody learns to survive it failing.

Full story in the article.