Slack's cache manager was too good at its job.

transcript

This part of Slack was objectively better. That made the outage worse. 22 February 2022. Just after 6 a.m. Pacific.

Slack only works after your app boots. It fetches your channels first. That morning, boots failed. A database was badly overloaded.

One question did much of the damage. Who's in your group DMs? Normally the cache answers. Fast and cheap. Miss the cache, and you must ask every shard of the database.

That day, Slack was upgrading Consul. Twice before, it went fine. Slow. One host at a time. Very careful. But each restart blinked a cache node out of the catalogue. Then peak traffic hit. And the cache hit rate started falling.

Here's the twist. The cache manager, Mcrib, did its job perfectly. Node gone? Swap in an empty spare. Node back? Flush it. Faster healing meant more churn. More churn meant colder cache.

Misses rise. Every shard gets hammered. Queries time out. So the cache can never refill. They paused the upgrade. Too late. It was a cascading failure.

So Slack throttled boots. Connected users could work again. They fixed the query to fetch only what was missing. And read replicas too. The caches refilled. Boots came back in small steps.

If your database can't survive without the cache, the cache is capacity. Change it slowly. Even when your tools can change it fast.

Full story in the article.