One routine resize, 17% of Discord's sessions gone, and why restarting made it worse

transcript

One routine change. 17 percent of Discord's sessions, gone at once. Voice and video were degraded for over three hours.

12:13 Pacific. Session pods get bigger and fewer in one zone. Clients reconnect. A per-host limit hadn't been re-tuned after the move to Kubernetes. More pods than VMs, so more got through. Gateways run out of memory. Then calls can't start. Awaiting Endpoint. So they restart every voice syncer. 13:05.

Calls get recreated. 15 voice syncers open a flood of connections. To over 25,000 SFUs. Through two single-process supervisors. One mailbox each. Even the etcd heartbeat queued behind the flood. Instances fell out of the ring.

The restart? Every instance re-created its calls at once. By 13:09, the mailboxes were growing again. What worked: cut demand first. Lower spawn limits, by 13:43. Then add supply. 15 new instances. Healthy by 14:26.

A big spike will always find a bottleneck. Cut demand first. Then add supply. And re-tune every per-host limit when the hosts change.

Full breakdown in the lecture.