Kafka was down for 2 hours. Not one order event was lost.

transcript

Kafka goes down for 2 hours. Orders keep pouring in. Not one order event is lost. Here's why. And what it still costs you.

Every order still commits, with its event, in one database transaction. The events just wait in the outbox table. Published at, still empty. The publisher can't reach Kafka. After 2 hours, a million rows are waiting. All safe.

But here's the catch. Safe isn't free. The table grows fast. Disk space alerts fire. Queries on the outbox slow down. And table bloat slows down your application's own writes.

First, fix Kafka. The publisher catches up on its own. It may take hours. To go faster: more publishers, bigger batches, a shorter poll interval. Disk about to run out? Push back. Return 503, service degraded. Better than losing events.

Long term, partition the outbox by creation time. Old partitions drop instantly. And alert on the oldest unpublished event. Over 5 minutes means nothing's leaving. Plus the count. Over 10,000 unpublished, someone should know.

With an outbox, a broker outage becomes a backlog. Not data loss. Your database is the buffer. Just watch how full it gets.

Full breakdown in the lecture.