Built for 5,000 a second. Hit with 200,000. Down in 10 minutes.

transcript

Black Friday, 2019. A flash sale alert goes out to 10 million users. The pipeline was built for 5,000 events a second. It gets 200,000.

30 seconds in, 2 million messages are waiting in the queue. One minute in, the consumers run out of memory. They crash. Rebalancing adds more lag. Two minutes in, every consumer is down. 5 million queued. The database pool runs dry. The API returns 503s. At 09:10, the whole site is down.

Why so fast? In a synchronous system, a busy server makes the client wait. It can't send faster than answers come back. A free brake. A queue removes that coupling. That's its whole point. So the producer never waits. The queue soaks up the difference. Until it can't.

It happens in three phases. Absorption: the queue buffers the gap. Doing its job. Exhaustion: depth in millions, memory climbing, lag growing to hours. Act now. Collapse: out of memory, crashes, timeouts cascading upstream. The point of no return.

The root cause? The system had no way to say: slow down, I can't keep up. So build one. Watch lag. Bound every queue. Then shed, rate limit, scale or spill.

A queue doesn't remove overload. It hides it, for a while. Async has no natural brake. You have to build one.

Full breakdown in the lecture.