One deploy, 10 million phones reconnecting at once

transcript

One routine deploy. 10 million phones reconnect at once. Your own users become the flood. Here's how to avoid it.

A chat backend has about 200 gateway servers. Each one holds about 50,000 open connections. Real phones, right now. 10 million in total. Restart a gateway, and its phones get cut off.

Start small. One gateway crashes. 50,000 clients drop. They reconnect to other gateways. With randomised delays, so not in one wave. Each one syncs from the last sequence it has. Nothing is lost. Every message was stored before the sender's tick.

But here's the catch. A deploy restarts every gateway. All 200 at once? 10 million clients reconnect at once. A thundering herd.

So never restart them all together. Drain a few gateways at a time. First, stop accepting new connections. Ask clients to reconnect elsewhere, with jitter. Then restart. Empty. Then the next few.

Stateless servers restart in any order. Stateful ones hold people. A few at a time. Never 10 million clients at once.

Full breakdown in the lecture.