A celebrity posts. One Kafka consumer does a million sends alone.

transcript

A celebrity with a million followers posts at 10 AM. One Kafka consumer does all the work. The others sit idle.

The naive way: notify all followers of user X. A million requests, instantly. One Kafka partition takes all the fan-out. The database gets a million preference and token lookups in a burst. A million pushes hit the provider's rate limits. The system drowns.

Why one partition? Look at the key. You keyed by the source user. The celebrity. Same key, same partition. One consumer does everything. Kafka happily accepts repeated keys. That's exactly how a hot partition forms. Key by the recipient's user ID instead. The fan-out spreads across every partition.

And don't block the post. Save a fan-out job, and return its ID. A worker pages through followers, 1,000 at a time. A token bucket holds it to 5,000 a second. Send, and checkpoint every 10,000. A crash doesn't mean starting over.

The celebrity caused the work. The recipients spread it. Key by who receives the message, not who caused it.

Full breakdown in the lecture.