A new song at 9 pm, and a partition on fire

transcript

9 pm in Jakarta. A big artist drops a song. A nightly model won't know it until tomorrow.

Here, its ID is admitted in seconds. Served within minutes. By 10, it's in 15 percent of the region's impressions. That's assumed. Then the alarm. The joiner falls 25 minutes behind. By midnight, about 2 million videos use the sound.

Why? Look at the feature log. It was partitioned by video ID. So every viral video lands on one partition. That partition overflows. The joiner lags.

A key that follows popularity makes a hot partition. Key by impression ID instead. One key per event. It spreads evenly, however viral the song gets.

Meanwhile, nothing is lost. Kafka buffers it. The trainer reads the newest data first. Batch picks up what it skipped.

And fans want the song in every slot? The rule holds. Never two in a row with the same sound.

Keys that follow popularity get hot. Key by the event.

Full breakdown in the lecture.