Half the designs in this course have a box labelled "Kafka" in the middle. This question asks you to open the box. How do you accept a million messages a second, keep them in order, never lose one that was acknowledged, and let a dozen different services read the same stream at their own pace? The answer is a surprisingly simple data structure, an append-only log, plus careful replication.
The Kafka-style log interview end to end: a log rather than a work queue, so billing, analytics and search each keep their own offset and can replay. A million 500-byte messages a second is 500 MB/s in, about 900 TB for 7 days at three replicas, around 50 brokers. Producers send keyed batches to a partition leader, followers fetch, the leader acks once every in-sync replica has the data, and consumers pull. Deep dives cover append-only segments, page cache and zero-copy reads, acks=all with min.insync.replicas=2, leader election with epochs, and delivery semantics set by when offsets are committed.
Phase 1: Requirements (5 minutes)#
"Is this a work queue, where each message is handled once and deleted, or a log that several consumers read independently?"
"A log. The orders stream is read by billing, analytics and search indexing, each at its own speed. And they sometimes need to replay yesterday."
"Then consumption doesn't delete anything: each consumer group just remembers how far it has read."
Work queue (RabbitMQ / SQS style)
- A message goes to one consumer and is deleted on ack
- Per-message acks, redelivery, priorities and delays are natural
- Hard to replay; another reader needs another queue
- Ordering weakens as consumers are added
Log (Kafka style)
- Messages are appended and kept for the retention period
- Each consumer group tracks its own offset: many readers, one copy
- Replay = move the offset back
- Order per partition; parallelism = number of partitions
- Create topics with N partitions and a replication factor
- Produce messages with an optional key; the same key always goes to the same partition
- Consume as part of a consumer group: each partition is read by one member of the group
- Commit offsets; resume after a restart; reset offsets to replay
- Retention by time (7 days) or, for keyed topics, compaction (keep the latest value per key)
- Throughput: 1M messages/s in, several times that out
- Durability: an acknowledged message survives the loss of any one broker
- Ordering: strict within a partition
- Latency: produce-to-consume in tens of milliseconds
- Availability: a broker failure moves leadership in seconds, without operator action
Phase 2: Estimation (5 minutes)#
- Inbound: 1M msgs/s × 500 B = 500 MB/s
- With replication factor 3, disk writes across the cluster: 1.5 GB/s
- Three consumer groups reading everything: 1.5 GB/s outbound
- Retention: 500 MB/s × 604,800 s (7 days) ≈ 302 TB of data, × 3 replicas ≈ ~900 TB of disk
- Brokers with ~20 TB of usable disk: 900 ÷ 20 ≈ 45, call it ~50 brokers with headroom
- Per broker: 1.5 GB/s ÷ 50 ≈ 30 MB/s of writes and about the same of reads: well within sequential disk and a 10 Gbps NIC
- Partitions: if one consumer handles ~5,000 msgs/s, the busiest topics need hundreds of partitions for parallelism
"Storage dominates the broker count, not throughput. And that only works because every write is sequential: a log is the friendliest thing you can do to a disk."
Phase 3: API (3 minutes)#
produce(topic, key, value, acks=all) -> (partition, offset)
partition = hash(key) mod partitions # same key → same partition → ordered
poll(group, topics, max_records) -> records from each assigned partition, from the group's offset
commit(group, partition, offset) -> ok # "I've processed everything before this offset"
seek(group, partition, offset | time) -> ok # replay
create_topic(name, partitions=48, replication=3, retention=7d | compact)Producers send in batches (say up to 64 KB or 10 ms, whichever comes first), compressed. Consumers pull. A slow consumer just falls behind; it never pushes back on the broker or other consumers.
Phase 4: High-level design (10 minutes)#
%% title: A distributed log: partitions replicated across brokers, a controller quorum, pulling consumers
flowchart TD
P@{ icon: "sd:servers", label: "Producers<br/>batch · key → partition" }
subgraph BR["Brokers"]
B1@{ icon: "sd:server", label: "Broker 1<br/>orders-0 leader · orders-1 follower" }
B2@{ icon: "sd:server", label: "Broker 2<br/>orders-1 leader · orders-0 follower" }
B3@{ icon: "sd:server", label: "Broker 3<br/>followers of both" }
end
CT["Controller quorum (Raft)<br/>topics · leaders · in-sync replicas"]:::core
G1@{ icon: "sd:worker", label: "Consumer group · billing" }
G2@{ icon: "sd:worker", label: "Consumer group · analytics" }
P -->|"produce to leader"| B1
P --> B2
B1 -.->|"followers fetch"| B2
B1 -.-> B3
CT -.->|"metadata · elections"| BR
B1 --> G1
B2 --> G1
B1 --> G2
class P client
class B1,B2,B3 store
class G1,G2 service- Route. The producer hashes the key (
order_42) to partitionorders-0and sends its batch to that partition's leader, Broker 1. - Append. The leader appends the batch to the end of the partition's current segment file and assigns offsets.
- Replicate. Followers on Brokers 2 and 3 fetch new data from the leader and append it to their copies.
- Acknowledge. With
acks=all, the leader replies once every in-sync replica has the batch. The message is now committed. - Consume. Each consumer group's member for
orders-0polls from its committed offset, processes the batch, and commits the new offset.
The controller is the brain, not the data path: it stores which brokers are alive, which broker leads each partition, and which replicas are in sync. It's a small Raft quorum, so it can't split-brain. (Kafka once used ZooKeeper for this and has since moved to a built-in Raft quorum.) Messages never flow through it.
Phase 5: Deep dives (15 minutes)#
Deep dive 1: Why an append-only log is so fast#
Segments
A partition is a directory of segment files, each up to ~1 GB. Writes only ever append to the newest segment. Old segments are immutable.
Sequential I/O and the page cache
Appending is sequential, the fastest thing a disk does. The broker writes to the OS page cache and lets the kernel flush; consumers that are caught up read recent data straight from memory.
A sparse offset index
Each segment has a small index: every few KB, "offset 1,204,000 is at byte 8,388,608". To read from any offset, binary-search the index, then scan a few KB.
Zero-copy reads
Sending stored bytes to a consumer uses sendfile, so data goes from page cache to socket without being copied through the broker process. Stored batches stay compressed end to end.
Retention is deleting files
Expiring 7-day-old data means deleting whole old segment files: no per-message deletes, no compaction of the remaining data. Compacted topics instead rewrite old segments keeping only the latest value per key.
Batching ties it together: one network request, one disk append and one replication fetch carry hundreds of messages. That's how a few dozen brokers handle a million messages a second.
Deep dive 2: Replication, and what "acknowledged" really means#
When is it safe to say "got it"?
A partition has 3 replicas. The producer sends a message and gets an ack. Then the leader's disk dies. Under what settings is the message guaranteed to survive, and what does that cost when a follower is slow?
One strong answer
acks=1: the leader acks after its own append. If it dies before followers fetched, the message is gone. Fast, not durable.acks=all+min.insync.replicas=2: the leader acks only when every in-sync replica has it, and refuses writes if fewer than 2 replicas are in sync. Any acknowledged message is on at least 2 brokers, so losing one can't lose it.- The cost: a follower that falls behind (slow disk, GC pause) is dropped from the in-sync set after a timeout, so it can't stall the partition forever. If too many drop out, the partition stops accepting writes rather than accept writes it can't protect: that's choosing consistency over availability.
Two more ideas complete the picture:
- The high watermark. Consumers can only read up to the last offset that all in-sync replicas have. That way no consumer ever reads a message that might vanish in a failover.
- Leader election. When a leader dies, the controller picks a new leader from the in-sync set, bumps the partition's leader epoch, and tells everyone. Writes stamped with an old epoch are rejected (fencing again). Allowing an out-of-sync replica to become leader ("unclean election") keeps the partition writable but loses data; it should be off for anything that matters.
Delivery semantics are decided at the edges:
| Where | Setting | Result |
|---|---|---|
| Producer retries | Idempotent producer (producer ID + sequence numbers) | A retried batch isn't appended twice |
| Consumer | Commit offset after processing | At-least-once: a crash means reprocessing a few messages |
| Consumer | Commit offset before processing | At-most-once: a crash means skipping a few |
| Read-process-write | Transactions: output + offset commit atomically | Exactly-once within the log |
Phase 6: Trade-offs and failure modes (5 minutes)#
A broker dies
Its partitions' leadership moves to in-sync followers in seconds. Producers and consumers refresh metadata and reconnect. A replacement broker re-replicates from the leaders.
A hot partition
One key (a celebrity's user ID) floods one partition, and only one consumer per group can read it. Choose keys with enough spread, or add a suffix to the hot key when strict per-key ordering isn't needed.
A slow consumer
It just lags; the log keeps its data for 7 days. Alert on consumer lag long before it approaches retention, or the consumer will lose data it never read.
Rebalancing storms
Adding a consumer reassigns partitions, pausing the group. Incremental (cooperative) rebalancing moves only the partitions that need to move.
Too many partitions
Each one costs file handles, memory and election time. Size partitions for target consumer parallelism, not "as many as possible".
| Decision | We chose | We gave up |
|---|---|---|
| Model | Log with consumer offsets | Per-message acks, priorities, delays |
| Durability | RF 3, acks=all, min in-sync 2 | A little latency; writes stop if 2 replicas are down |
| Ordering | Per partition, by key | Global order across a topic |
| Metadata | Raft controller quorum | A simpler single controller |
Go deeper in the course#
- Week 3, Day 1: Queue vs stream: the queue-or-log decision, partitions, consumer groups and offsets
- Week 3, Day 3: Backpressure and flow control: why pull-based consumers and lag monitoring matter
- Week 3, Day 4: Dead letters and poison pills: what a consumer does with a message it can't process
- Week 3, Day 2: Transactional outbox: getting a database change into the log exactly once
- Week 1, Day 2: Replication trade-offs: leaders, followers, sync vs async replication and failover
- Week 5, Day 5: Leader election and coordination: consensus, coordination services and fencing, the controller's job
- Week 2, Day 2: Idempotency in practice: idempotent producers and consumers
- Week 1, Day 4: Hot keys and skew: the hot-partition problem
- Week 3 capstone: event-driven order system: a whole system built around the log
Check yourself#
Distributed message queue
Q1/4 Billing, analytics and search all need every order event, at their own pace. Why does a log fit better than a work queue?
Answer & explanation
Answer: A
Consuming doesn't delete anything in a log; each consumer group just tracks how far it has read, and can rewind to replay.
Q2/4 With replication factor 3, which setting guarantees an acknowledged message survives one broker's loss?
Answer & explanation
Answer: C
acks=all waits for every in-sync replica, and the minimum of 2 means an ack always implies at least two copies.
Q3/4 A consumer commits its offset before processing a batch, then crashes mid-batch. What happens?
Answer & explanation
Answer: B
The committed offset says "done", so the restarted consumer resumes after the batch. Commit after processing for at-least-once.
Q4/4 Why can a few dozen brokers handle a million messages a second?
Answer & explanation
Answer: D
The log turns every write into a sequential append and every catch-up read into a page-cache hit sent with sendfile.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.