"Just use Kafka." Ask one question first.

transcript

Someone in the design review says: just use Kafka. Before you nod, ask one question. Is this work to be done, or things that happened?

A queue is a post office box. You drop a letter. One person picks it up. Once it's picked up, it's gone. You can't read it again. A stream is a newspaper archive. Today's paper is added. Anyone can read any past edition. The papers stay, until a cleanup policy removes them. In Kafka, that's retention.

In a stream, events stay in the log after they're read. Orders, analytics and audit all read the same events. Each tracks its own offset. Analytics wants to rerun last week with new logic? Rewind its offset. Orders don't even notice.

So ask three questions. Do you need to replay messages? Stream. Do several systems need the same data? Stream. Is it a job to be processed once? Queue.

LinkedIn's activity and Uber's trips? Streams. Many readers, replay, ordering. Delivering webhooks, encoding videos? Jobs for workers. Queues. And if you're unsure? Start with a queue. It's simpler, and you can migrate.

Neither is better. They answer different questions. Queues distribute work. Streams keep history.

Full breakdown in the lecture.