10 million open sockets on 200 servers: which one is Bob on?

transcript

Alice sends Bob a message. Bob is connected to one of 200 servers. Which one? And the whole trip should take under 500 milliseconds.

Assume 10 million people online at peak. Each holds an open WebSocket. One server comfortably holds about 50,000 connections. So about 200 gateway servers. Each socket lives on exactly one.

Here's the catch. A normal request can go to any server. A message for Bob must go to his server. The load balancer doesn't know which.

So register on connect. Bob's laptop joins gateway 2. Gateway 2 writes: Bob, laptop, gateway 2. With a TTL. Alice's message reaches the chat service. It asks: where are Bob's devices? Laptop on gateway 2. Forward it there. Phone on gateway 7. Forward it there too. Every device gets it.

But what if gateway 2 crashes? The registry still says Bob is there. Not for long. Heartbeats refresh each entry's TTL. A dead gateway stops refreshing. Its entries simply expire. The registry never points at a dead gateway for long.

No broadcast to all 200 servers. One lookup, then straight to Bob. Stateful gateways, a registry with a TTL. Everything else stays stateless.

Full breakdown in the lecture.