One malformed field froze checkout for 4 hours

transcript

2 AM. Your phone goes off. The order consumer has been crash-looping for 4 hours. 50,000 messages stuck behind it. Customers can't check out.

The cause? One order event. Its items field is a string. It should be a list. One malformed field.

The consumer reads it. Throws an exception. Crashes. Kubernetes restarts it. And what does it fetch? The same message. Crash, restart, crash. About 50 times in 4 hours. Kubernetes backs off to one restart every 5 minutes. By the time on-call wakes up: 847,000 dollars of lost revenue. From one message.

But why is everything else stuck? The offset never moves past the bad message. So every later message in that partition waits behind it. That's a poison pill.

The fix: a dead letter queue. Normal messages flow through as usual. A message that can't be processed moves to the DLQ. And the consumer moves on. Later, someone inspects it, fixes the cause, and replays it. Or discards it.

One bad message should cost you one message. Not your whole pipeline. Quarantine it. Then move on.

Full breakdown in the lecture.