A routine upgrade emptied Slack's caches, and timeouts kept them empty

transcript

A routine upgrade. 25 percent of the fleet. What could go wrong? It tipped Slack into overload.

22 February 2022. A Consul agent upgrade restarts agents on cache nodes. The restart caused cache nodes to drop out of discovery. Mcrib, the cache control plane, swaps in empty spares. Nodes that rejoin? Flushed. The hit rate falls.

Boots ask: who's in this group DM? Cache miss. And the table is sharded by user. So the query has to hit every shard. Shards slow down. Queries time out.

Here's the trap. A timed-out query can't refill the cache. So misses stay misses. Load climbs faster than the miss rate. It holds Slack down even after the trigger stops.

Slack throttled boots. Read only what was missing from memcached. Used replicas. Later, it moved the query to a table sharded by channel.

Can your database survive without the cache? If not: That cache isn't an optimisation. It's capacity.

Full breakdown in the lecture.