Google takes its Chubby lock service down on purpose when it has been too reliable.

Chubby is Google's lock service. Its 2006 paper describes it as providing "coarse-grained locking as well as reliable (though low-volume) storage for a loosely-coupled distributed system": a small, trusted place where many systems coordinate. Deployments in the paper served a few tens of thousands of clients at once. The global instance of Chubby puts each of its replicas in a different geographical region.

That made global Chubby very reliable, and that turned out to be the problem. In the SLO chapter of Google's Site Reliability Engineering book, Marc Alvidrez explains that true global Chubby outages were so infrequent that service owners began adding dependencies on it "assuming that it would never go down". When it did go down, however rarely, those services couldn't function properly. Chubby's failures "consistently generated service outages, many of which were visible to end users". Its high reliability had provided "a false sense of security".

Google's fix was not to make Chubby even more reliable. SRE makes sure global Chubby meets its service level objective (SLO), its availability target, but "does not significantly exceed" it. In any given quarter, if a true failure has not dropped availability below the target, SRE synthesises a controlled outage by intentionally taking the system down.

That flushes out unreasonable dependencies on Chubby shortly after they are added, and, in the book's words, "forces service owners to reckon with the reality of distributed systems sooner rather than later". The same chapter warns that without an explicit SLO, users form their own beliefs about a service's availability, which can lead to over-reliance on it.

Why it matters: People build on how a service really behaves, not on what you promise. If a dependency never fails, nobody learns to survive its failure. Spending the allowed downtime on purpose keeps everyone honest.