Day 1Queue vs Stream

Work to do vs things that happened. A queue holds jobs: each message goes to one worker and is deleted after the ack. A stream is an append-only log: events stay, each reader tracks its own offset and can replay.

Consumer groups. Kafka's model: every group gets all messages (fan-out) and within a group each partition goes to exactly one consumer (load balancing). Parallelism equals the partition count.

Order lives in the partition. Kafka orders messages only within a partition. Key by the entity that needs order (order_id, user_id) so "created, paid, shipped" can't be processed out of order.

At-least-once plus idempotent consumers. Exactly-once needs special support on both sides, so most systems ack after processing and dedupe, e.g. by partition and offset stored in the same transaction as the work.

• Kafka for simple background jobs that a queue with built-in retry would handle.
• No partition key, so one order's events land on different partitions.
• One partition "to start": only one consumer can ever work on it.

Day 2Transactional Outbox

The dual-write problem. Saving an order then publishing to Kafka are two writes with no shared transaction. A crash in between leaves an order no downstream service hears about; flipping them publishes events for orders that never saved.

Write the event to the database. Insert the business row and an outbox row in the same transaction: both commit or neither does. A separate publisher reads unpublished outbox rows and sends them to Kafka.

Polling or CDC. A poller queries published_at IS NULL every few seconds: simple, but adds latency and database load. CDC (Debezium) tails the write-ahead log: ~10–100 ms latency, no polling, more to operate.

Still at-least-once. A publisher that crashes after Kafka acks but before marking the row published sends it again. Consumers dedupe by event ID; Kafka transactions can't cover the database update.

• Writing the outbox row in a separate transaction: the dual write is back.
• Publishing to Kafka inside the request path, so Kafka trouble fails user requests.
• No alert on outbox lag: the table grows to 10M rows before anyone notices.

Day 3Backpressure and Flow Control

Async removes natural backpressure. A synchronous client can't send faster than the server answers. A queue absorbs the gap instead, until memory runs out: absorption, then resource exhaustion, then collapse.

Watch lag, depth and latency. Consumer lag (alert at 10,000 messages or 5 minutes), queue depth (100,000 or growing for 5 minutes) and queue wait time. Throughput falling while lag grows means consumers are struggling.

Four responses. Shed low-priority load (metrics, logs), rate-limit producers (adaptively, as lag grows), scale consumers (Kafka: up to the partition count), or spill overflow to disk or S3 for later.

You get two of three. Completeness, latency, stability: pick two. Most systems keep completeness and stability and accept higher latency during spikes; real-time pricing drops stale events instead.

Recover gradually. After an outage, don't jump from 100 to 10,000 messages a second: the backlog floods downstream. Step up (500, 1,000, 2,500, 5,000, 10,000) after a minute of stability each.

• Unbounded in-memory queues: the OOM killer takes every message with it.
• Rate limiting only at the API gateway while internal services overwhelm the database.
• Hot partition keys: one consumer drowns while the others idle.

Day 4Dead Letters and Poison Pills

A poison pill blocks everything behind it. A message that crashes the consumer every time causes a crash loop: restart, fetch the same message, crash. In the lesson, one bad message held 50,000 orders for 4 hours.

Classify the failure. Transient (timeouts, rate limits): retry with backoff. Permanent (bad JSON, missing field, rule violation): straight to the DLQ. Poison pills (crashes, hangs, memory blowups): quarantine immediately.

Delayed retry topics. Instead of retrying in place, republish to retry topics with growing delays (1 s, 10 s, 60 s), then the DLQ. The main topic keeps flowing; Uber described this design for Kafka.

A DLQ is an operational tool. Store the original message, error and stack trace, retry count, timestamps and source topic, partition and offset. Monitor depth, growth and age, triage daily, and build replay tooling from day one.

Replay needs idempotency. Check the payment_id before reprocessing so a replay can't double-charge. Old messages may not match today's schema: version them, migrate on replay, or close them out.

• Resetting the retry count to 0 in each consumer instead of reading it from headers: infinite loop.
• A DLQ nobody monitors: it becomes a graveyard of lost data.
• A DLQ consumer that processes instead of just storing and alerting.

Day 5Audit Log System

Who did what, to which resource, when, from where. An audit log serves auditors, legal and security, not engineers: strict schema, retention in years, and it must be immutable. An application log is for debugging and can be deleted.

Capture in the same transaction. Write the audit event to an outbox in the same transaction as the business action, so a transfer can't succeed without its record. Kafka, consumers and the DLQ carry it asynchronously.

Immutable and tamper-evident. No UPDATE or DELETE: corrections are new events. Each event carries the previous event's hash, checkpointed somewhere an attacker can't change, and cold files sit under S3 Object Lock.

Tiered storage. Hot (0–90 days) in TimescaleDB for fast investigation queries, warm (to 1 year) in ClickHouse or BigQuery, cold (1–7 years) as Parquet on S3, queryable with Athena.

Never shed audit events. When consumers fall behind, scale them, batch harder and alert, but accept latency: audit data can't be dropped. Sort investigations by ingestion time and keep both timestamps.

• Writing the audit event after the business transaction: money moves with no trail.
• "Fixing" audit rows with UPDATE, which breaks the chain of custody.
• Logging only "delete doc_123" without actor, context or reason.

BossEvent-Driven Food Delivery Order System

Size the dinner rush. 500,000 orders a day, 60% in 6–9 PM: ~28 orders/s, ~56 at the peak, design for 100. With ~8 events per order and 7 consumer services, that's 5,600 event deliveries a second.

Kafka keyed by order_id. Seven services need the same events, replay helps debugging, and each order's events must stay in order. Partition orders.events (32 partitions) by order_id; each service is its own consumer group.

Different consumers, different guarantees. Payment needs effectively-once with an idempotency key per order (Stripe won't charge twice). Analytics can lag, push is fire-and-forget, fraud detection needs real time.

Retry topics, then the DLQ. Transient failures go to retry topics delayed 1, 5 and 15 minutes; after 3 retries, the DLQ. Payment failures retry then notify the customer; restaurant rejections find an alternative or refund.

Audit for disputes. 2% of orders are disputed. Every service writes immutable audit events, so support queries one order's full timeline in minutes instead of correlating logs across services.

• Saving the order and publishing to Kafka as two writes: lost or phantom orders.
• Shedding driver-assignment events: a late driver hurts more than late analytics.
• Retrying forever in the main topic, so one failing order blocks the rest.