The login code that waited behind 9.2 million cashback pushes

transcript

Rohan asks for a login code. 40 seconds past 8. It arrives 8 minutes later. Already expired.

8 PM sharp. Marketing drops 10 million cashback pushes into the queue. Forty seconds later, Rohan's OTP joins. Same queue. Behind them all. He taps resend three times. Each tap buys another SMS. At 8 minutes past 8, the code leaves. Its 5 minutes are up.

The workers drain 20,000 messages a second. In 40 seconds, that's 800,000. So 9.2 million are still ahead. 9.2 million, at 20,000 a second. That's 460 seconds of waiting.

So just add workers, right? Here's the catch. Draining 10 million in a minute needs 167,000 a second. That's over 8 times the fleet. For an eight-minute burst.

The real fix is structural. Classes don't share a queue. OTPs get a critical lane, with their own workers. Payments ride the transactional lane. And campaigns are paced into a bulk lane.

A campaign can wait a minute. A login code can't. So give them different lanes.

Full breakdown in the lecture.