Just after 6 a.m. Pacific Time on 22 February 2022, users started reporting that Slack wouldn't connect. Slack only works once your app has "booted": fetched your channels and recent conversations. Boots often failed, and a database holding channel membership was badly overloaded.
Much of that load came from one query: the list of people in your group direct messages. The data lived in a database split into shards by user, so finding the members of one conversation meant asking every shard. Normally the answer sat in Memcached, Slack's cache. With much of the cache gone, nearly every boot had to query every shard.
The cache was draining because of maintenance. Slack was upgrading Consul, its service-discovery system, on 25% of the fleet that day, after two earlier 25% steps went fine the week before. Agents restarted slowly, one host at a time. But a restarting agent briefly takes its host out of the catalogue, and a newer component, Mcrib, quickly swapped in an empty spare cache node and flushed the old one when it came back. As the restarts met the day's peak traffic, the cache hit rate fell, database queries timed out, and the cache couldn't refill.
Pausing the upgrade didn't help: the system was stuck in a cascading failure. Slack throttled app boots so already-connected users could work, changed the query to fetch only the data missing from the cache, and let it read from database replicas too. Caches refilled, and boots were let back in small steps.
The irony, in Slack's own words: Mcrib was "objectively a better system", but "its efficiency made the broader system behave in a less safe way."
- Started just after 6 a.m. Pacific, 22 February 2022
- 25% of the fleet upgraded in that step, after two earlier 25% steps
- 1 cache miss meant a query to every shard of the database
Why it matters: A cache that the database can't survive without is part of your capacity, not an optimisation. Change cache membership slowly, and avoid queries that fan out to every shard.