Slack's first workday of 2021 overloaded a network hub that was meant to scale itself.

Over the holidays, people stop working and Slack gets quiet. On the first Monday back, apps on people's devices have cold caches and pull down more data than usual. Slack's engineers wrote that they go "from our quietest time of the whole year to one of our biggest days quite literally overnight".

Slack runs its services in many separate private networks on AWS, linked through hubs called Transit Gateways. AWS manages them, and they are meant to scale automatically. On 4 January 2021, one of them didn't scale fast enough. It became overloaded and started dropping packets.

During the Americas' morning, an external monitor paged Slack: errors were creeping up. Then Slack's own dashboards and alerts went down, because they sat in a different network from their databases and depended on the same gateways. Engineers had to debug without their usual dashboards.

At 6:57 a.m. PST, 99% of messages were still being sent; normally it's over 99.999%. Then the usual burst of traffic at 7 a.m. hit the slowed network, and Slack became unavailable. Automation made it messier. Servers that couldn't reach their backends were marked unhealthy, and Slack's systems tried to replace them. A drop in CPU use made autoscaling shrink the web tier, ending engineers' sessions on the machines they were investigating. Then Slack tried to add 1,200 servers in 14 minutes. The service that sets up new servers, itself on the bad network, hit the Linux open files limit and an AWS quota. Most new servers weren't fully set up and served nothing.

That service was back around 8:15 a.m. By about 9:15, Slack was slow but up again. AWS engineers, alerted by their own monitoring, raised the gateway's capacity by hand; by 10:40 a.m. the network was back to normal. Slack set itself a reminder, in Slack, to ask for the gateways to be scaled up ahead of time after the next holidays.

Why it matters: "Scales automatically" has a speed limit. If you know a huge jump is coming, scale ahead of it, and keep your monitoring off the paths it is meant to watch.