One routine deploy. 17% of Discord gone.

transcript

25 March 2026. Discord ships a routine deploy. And 17 percent of its sessions stop at once. And that wasn't the worst part.

Half of one zone's pods stop before handing off their sessions. So every client does what it was built to do. Reconnect. A rate limit lets too many through. The gateways run out of memory. That kills sockets that were fine. So they reconnect too.

Here's the trap. Everyone reconnects at once. The gateways overload, and health checks fail. Healthy sockets drop. And they reconnect too.

The flood reached Discord's voice control plane. Calls showed Awaiting Endpoint. For over three hours.

A fixed retry delay just moves the wave. Instead, wait a random time, with a doubling cap. Let the server say: retry later. And resume, don't reboot. And drain deploys slowly, a trickle at a time.

The trigger wasn't a surge in messages. It was everyone coming back at the same moment.

Full breakdown in the lecture.