Slack's first workday of 2021 broke a network hub meant to scale itself

transcript

Slack's quietest time of the year. Then one of its busiest. Overnight. 4 January 2021, part of the network couldn't keep up. The part built to scale itself.

First Monday back. Apps wake up with cold caches, pulling extra data. Slack's networks on AWS link through hubs called Transit Gateways. AWS manages them. They're meant to scale automatically. This one didn't scale fast enough. It started dropping packets. And Slack's dashboards? They depended on the very same gateways.

Morning. An outside monitor pages Slack. Errors are creeping up. Then Slack's own dashboards and alerts go down. Engineers debug without them. 6:57 a.m. 99 percent of messages sent. Normally? Over 99 point 999. 7 a.m. The usual traffic burst hits. Slack goes down.

But here's the catch. Slack's own automation made it messier. Servers that couldn't reach backends got marked unhealthy. And replaced. CPU use dropped, so autoscaling shrank the web tier. Engineers got cut off mid-debug. Then Slack tried to add 1,200 servers. In 14 minutes.

The service that sets up new servers? On the same bad network. It hit the Linux open files limit. And an AWS quota. Most new servers were never fully set up. They served nothing.

Around 8:15, the setup service works again. By about 9:15, Slack is slow, but up. AWS raised the gateway's capacity by hand. 10:40, back to normal.

Slack set a reminder, in Slack. Get the gateways scaled up ahead of time. Know a huge jump is coming? Scale ahead of it. Because scales automatically has a speed limit.

Full breakdown in the lecture.