Part 1System Design Fundamentals

HLD vs LLD. High-level design is the what: components, how they interact and why. Low-level design is the how: schemas, API contracts, algorithms, error handling. Interviews start high-level, then deep-dive one or two areas.

The four-step framework. Clarify requirements (3–5 min), estimate scale (5–7), draw the high-level design (15–20), then deep dive (10–15): schema, API, critical algorithms, edge cases and failures.

Functional vs non-functional. Functional requirements say what the system does; non-functional ones say how well: uptime, latency, throughput, consistency. The non-functional ones drive architecture, technology and cost.

Sync vs async. Sync is simple with immediate feedback, but the client blocks and services couple. Async through a queue decouples, absorbs spikes and allows retries, at the cost of eventual consistency and harder debugging.

API styles. REST for CRUD and public APIs, GraphQL when clients ask for exactly the data they need, gRPC (binary Protocol Buffers) for fast internal calls, WebSocket for server push.

• Designing before clarifying scope, scale and the read/write ratio.
• Skipping non-functional requirements: they decide the architecture.
• Diving into low-level detail before the high-level picture is agreed.

Part 2Infrastructure Building Blocks

CDN. Edge servers (PoPs) serve content near users: ~5–10 ms from a Mumbai PoP versus ~150–200 ms to a US origin. Cache static assets and identical-for-everyone responses; never user-specific data.

Versioned URLs. Put a content hash in the filename (app.a1b2c3d4.js) and cache it for a year as immutable. A deploy changes the URL, so users get fresh files with zero purge delay.

no-cache doesn't mean "don't cache". no-cache caches but revalidates (ETag, 304 Not Modified); no-store never caches; private keeps it out of the CDN; s-maxage gives the CDN its own TTL.

L4 vs L7 load balancing. L4 sees IP and port: very fast (<100 μs added), any protocol. L7 sees the whole HTTP request: routes by path, host, header or cookie and terminates TLS, adding 1–5 ms.

Shallow vs deep health checks. Point the load balancer at a shallow liveness check. A deep check that queries the database marks every server unhealthy when the DB is down; use deep checks for monitoring.

Stateless servers scale. Keep no session in server memory: sessions in Redis, data in PostgreSQL, files in S3. Then any server takes any request and a crash loses nothing.

• Caching user-specific or authenticated responses at the CDN.
• A deep health check behind the load balancer: one database outage removes every server.
• Microservices for a small team with unclear requirements; start with a monolith.

Part 3Back-of-the-Envelope Estimation

Per day to per second. A day is 86,400 seconds; round to 100,000. So X per day ÷ 100,000 ≈ X per second: 1 billion a day is about 10,000 a second.

The four estimates. Traffic (RPS, read/write split, peak), storage (objects × size × retention × replication), bandwidth (RPS × payload, ingress and egress) and infrastructure (servers, replicas, cache).

Latency ratios matter. L1 cache 0.5 ns, memory 100 ns, SSD random read 150 μs, HDD seek 10 ms; datacenter round trip 0.5 ms, cross-continent 150 ms. Cache aggressively, batch, and minimize round trips.

Media dominates storage. For a Twitter-like service, text is 30 GB a day but images and videos add ~7 TB a day: media is 99%+ of storage. Egress for images is why the CDN is essential.

Formulas to say out loud. RPS = daily actions ÷ 100,000. API servers = peak RPS ÷ 5,000 ÷ 0.7. Read replicas = read RPS ÷ 30,000. Cache = hot data: 20% of the total, or the last N days.

• Forgetting growth: 100 GB now can be 1 TB in a year at 10x growth.
• Forgetting replication (500 GB × 3 = 1.5 TB) or that images make posts 100x bigger.
• Mixing megabytes and megabits: MB/s × 8 = Mbps.

Part 4Networking Fundamentals for System Design

TCP guarantees, at a price. TCP gives reliable, ordered delivery with flow and congestion control, but the handshake costs 1 RTT before the first request and one lost packet blocks everything behind it.

UDP when late data is useless. No handshake, an 8-byte header, message boundaries kept, no reliability. Use it for real-time video, voice, gaming, DNS and as QUIC's base.

HTTP/1.1 → 2 → 3. HTTP/1.1 sends one request at a time per connection. HTTP/2 multiplexes streams and compresses headers over one TCP connection. HTTP/3 runs QUIC over UDP: a lost packet only blocks its own stream.

Reuse connections. Each new connection pays TCP and TLS handshakes and starts in slow start. Keep-alive and connection pools avoid that; short connections also pile up in TIME_WAIT and exhaust ephemeral ports.

Scaling WebSockets. Connections are stateful, so the load balancer needs stickiness, and users on different servers reach each other through a pub/sub backbone (Redis, Kafka). Use SSE for one-way server push.

• Opening a new connection per request instead of pooling: handshakes, slow start, port exhaustion.
• WebSocket for request-response or one-way push, where HTTP or SSE is simpler.
• Assuming HTTP/2 solved head-of-line blocking: it still rides one TCP connection.

Part 5Operating System Concepts for System Design

Processes isolate, threads share. A process owns its memory and file descriptors, so one crash kills only itself. Threads share the heap and descriptors with their own stacks: cheap to create and switch, but one crash kills them all.

Event loops scale connections. Thread-per-connection doesn't scale past ~10K connections. epoll registers descriptors once and returns only the ready ones, so one thread serves thousands (Node.js, nginx, Redis).

Context switches cost more than they look. Saving registers is cheap; the TLB flush, cold CPU caches and pipeline flush are not. Use async I/O, thread pools, batching and CPU affinity to switch less.

Write-ahead logging. Append the change to a sequential log and fsync it before touching data files; on a crash, replay the log. Sequential HDD writes run ~100 MB/s versus ~1 MB/s random.

Containers are namespaces plus cgroups. Namespaces isolate what a process sees (PID, network, mounts, users); cgroups limit what it uses (CPU quota, memory). Exceed the memory limit and the container is OOM killed.

• Thread-per-connection for a million connections instead of an event loop.
• Taking locks in different orders across threads: a deadlock waiting to happen.
• Running a database that caches pages itself without O_DIRECT, so data is cached twice.

Part 6System Design Terminology & Theory

CAP is C or A during a partition. Partitions will happen, so P isn't optional. During one, a CP system refuses what it can't confirm (ZooKeeper, etcd); an AP system answers with possibly stale data (Cassandra, DNS).

PACELC. Even with no partition, strong consistency costs latency: synchronous replication waits for the slowest replica. DynamoDB and Cassandra are PA/EL; HBase is PC/EC.

Isolation levels. Read uncommitted allows dirty reads; read committed (PostgreSQL's default) allows non-repeatable reads; repeatable read (MySQL's default) allows phantoms; serializable allows none.

Consistency models. Linearizability respects real-time order; sequential keeps only each process's program order; causal keeps cause before effect (vector clocks); eventual only promises convergence.

Delivery semantics. At-most-once may lose messages; at-least-once may duplicate them, so consumers must be idempotent; exactly-once needs idempotent producers and consumers, or transactional messaging.

Error budgets. A 99.9% SLO allows 43.2 minutes of downtime in a 30-day month. Budget left: ship fast. Budget gone: freeze features and fix reliability. Keep the SLA looser than the SLO.

• "Pick 2 of 3" in CAP: partition tolerance is a given, not a choice.
• Assuming exactly-once delivery is free instead of making consumers idempotent.
• Saga compensations that aren't idempotent: they may run more than once.

Day 1Partitioning Deep-Dive

Partitioning is how you scale writes. Vertical scaling hits a ceiling and read replicas don't fix write bottlenecks. Partitioning (sharding) splits data across machines, each owning a subset, so storage, throughput and memory add up.

Hash partitioning. hash(key) % N spreads keys evenly with an O(1) lookup, but range queries become scatter-gather across every partition, and changing N moves most keys.

Range partitioning. Contiguous key ranges keep a range query on one partition, but auto-increment IDs or timestamps send every new write to the last partition.

Directory partitioning. A lookup table maps keys to partitions: any key can go anywhere and rebalancing is a directory update, but the directory is a single point of failure and adds a lookup to every request.

Consistent hashing. Keys and nodes sit on a ring; a key belongs to the first node clockwise. Adding a node moves only about 1/(N+1) of keys, from one neighbour. Virtual nodes even out the load.

• Partitioning by hash(user_id + timestamp): perfect spread, but no more queries by user_id alone.
• Range partitions on auto-increment IDs or time, so every new write lands on one partition.
• Over-partitioning early: 1000 partitions × 100 app servers = 100,000 connections.

Day 2Replication Trade-offs

Three reasons to replicate. Replicas buy high availability (fail over to a copy), read scaling (reads spread across copies; writes don't) and geographic latency (read from a nearby copy).

Sync, async, semi-sync. Sync waits for followers: no loss, slower writes. Async acks at once: fast, but a dead leader loses unreplicated writes. Semi-sync waits for one follower: the production middle ground.

Quorums. With N replicas, if W + R > N every read overlaps the latest write. N=3 with W=2, R=2 is strongly consistent; W=1, R=1 is faster but only eventual.

Replication lag anomalies. Lagging followers break read-your-writes (your update vanishes), monotonic reads (data goes back in time) and consistent prefix (a reply shows before its message).

Split brain needs fencing. A network partition can leave two leaders accepting writes. Fence the old one: STONITH, storage or network fencing, or a lease it must keep renewing in ZooKeeper or etcd.

• Serving a balance, inventory count or permission check from an async replica.
• Failing over without fencing the old primary: two primaries, conflicting data.
• Last-write-wins across leaders: clocks disagree and writes silently vanish.

Day 3Rate Limiting at Scale

Limits must be counted together. With 50 servers each seeing a slice of a user's traffic, uncoordinated local counters let 100 requests through a 10-per-minute limit. Counts must be shared, synced or sticky.

Fixed window's boundary burst. One counter per window is simple and O(1), but a user can spend a full quota just before and just after the boundary: 2x the limit within seconds.

Sliding window counter. Weight the previous window by its overlap and add the current one: (8 × 75%) + 3 = 9. Two counters per key, no boundary burst; the default for API limits.

Token bucket allows controlled bursts. Tokens refill at a fixed rate up to a capacity and each request spends one: the long-term rate is bounded, bursts up to capacity pass. Compute refill from elapsed time.

Fail open or fail closed. When the limit store dies, over-allow user-facing and paid APIs; under-allow DDoS protection, cost control and login. A conservative local fallback sits in between.

• Each server counting alone: users spread requests and multiply the limit.
• One blanket fail-open rule: an AI call costing $0.01 each should fail closed.
• Leaky bucket for user-facing limits: it queues requests and adds latency even with spare capacity.

Day 4Hot Keys and Skew

Traffic follows Zipf's law. The nth most popular key gets about 1/n of the top key's traffic. With 1,000 keys the top 1% take ~40%; viral content pushes the top 0.01% past 90%.

Detect with a Count-Min Sketch. Fixed memory, O(depth) per update: hash the key into several counters and read the minimum. It may overestimate but never underestimates, which is all hot key detection needs.

Layer the defenses for hot reads. CDN edge, then local app memory, then Redis with split keys and request coalescing; the database sees only misses. In the lesson, traffic past the edge drops from 10,000 to ~2,800 req/s.

Write-heavy hot keys can't be cached. Buffer increments and flush periodically, shard the counter across N keys (reads sum all N), or use a G-Counter CRDT where each node owns its own count.

Bifurcate by popularity. Celebrities get their own code path: posts from accounts under 10K followers are pushed into feeds; celebrity posts are pulled at read time and merged. Show approximate counts like "1.2M likes".

• Assuming uniform traffic: one viral key turns a partition into a single-node bottleneck.
• Letting a hot key expire all at once: thousands of requests stampede the database.
• Splitting keys that are written often: every write fans out to all splits.

Day 5Session Store Design

Sessions are read-heavy and forgiving. Every request validates a session (80% reads), so latency must stay under 5 ms P99. Durability is best effort: a lost session means logging in again, inconvenient but not catastrophic.

Sticky vs distributed sessions. Sticky sessions keep state in one server's memory: fastest, but a server failure logs its users out and deploys lose sessions. A distributed store (Redis) costs a network hop and survives both.

Client-side sessions can't be revoked. A JWT or encrypted cookie needs no storage and scales anywhere, but you can't invalidate it before expiry, it's capped at 4 KB and rides on every request.

Region-affine sessions. Encode the home region in the session ID (sess_east_abc123) and route there. If a region dies, its users re-authenticate; make that painless with SSO or remember-me.

Session security basics. 256-bit random IDs, a fresh ID on login to block fixation, and cookies set HttpOnly, Secure and SameSite=Lax.

• Local session caches without short TTLs or pub/sub invalidation: logged-out sessions keep working.
• A user_sessions set without the sessions' TTL: it fills with expired IDs.
• Sequential, guessable session IDs instead of secure random ones.

BossDesign a URL Shortener at Scale

Clarify, then estimate. Ask before designing: spikes, code alphabet, duplicate URLs, analytics freshness, API rate limits. The answers set the target: 120,000 redirects/s during viral spikes.

Pre-allocated key ranges. A key service hands each API server a range of 10,000 counter values, scrambled and base-62 encoded to 7 characters. Only range allocation needs coordination; a crash just loses a few thousand codes.

Split redirect from creation. Redirects are 10x writes and need 99.99% uptime, creation 99.9%. Separate services scale and fail independently, and clicks go to Kafka asynchronously so counting never slows a redirect.

Hybrid sharding. URLs live on shards chosen by consistent hashing on short_code (fast redirects); a light user_urls index sharded by user_id serves "my URLs". Every create writes both.

• Auto-increment codes (predictable), random-then-check (races) or UUIDs (36 characters, not 7).
• A bare 301 redirect: browsers cache it forever, hiding clicks and ignoring deletes.
• Caching URLs without their expiry, so expired links keep redirecting.

Day 1Timeout Hell

Slow is worse than down. A down service fails fast; a slow one holds your threads. With 100 threads, a 10 s downstream cuts you from 1,000 to 10 requests a second. Timeouts turn slow into down.

Percentiles, not averages. Set a timeout at the ~65 ms average and 5% of healthy requests fail. Rule of thumb: timeout = P99 × 2–3, so P99 = 200 ms gives 400–600 ms.

Timeout budgets. Give each dependency a share of the SLA, not the whole SLA. For a 5 s payment: 4.5 s budget, fraud check 500–600 ms, bank 3.5 s, notification optional with ~300 ms.

Deadline propagation. Pass the absolute deadline down the chain, so each hop waits only for the time left. Per-hop 5 s timeouts across three services can add up to 15 s after the user gave up at 5.

Connection vs read timeout. Connection timeout covers the TCP handshake and should be short (1–5 s): if you can't connect, something is badly wrong. Read timeout covers the work and depends on the operation.

• No timeout, or 30 s "to be safe": threads pile up and you look dead.
• Timeout = average or P99, failing healthy requests every day.
• Adaptive timeouts without alerting on the timeout value and a hard maximum.

Day 2Idempotency in Practice

Request lost or response lost? A client can't tell them apart: both look like a timeout. No timeout tuning fixes that; the only answer is making the operation safe to retry.

The idempotency key. The client sends a unique key per logical operation. The server records it with status "processing" before calling the bank, stores the result, and returns the cached result for any repeat.

Pass the key downstream. Send your idempotency key to the bank too: it deduplicates as well, and on an ambiguous timeout you can ask it what happened to that key, then update your record.

Guard the edge cases. Hash the request body so a reused key with a different amount is rejected. Only completed requests block retries; a failed one may run again. Extend the TTL on completion.

• POST payments with no idempotency key: a double-click charges twice.
• A TTL shorter than the retry window, so a late retry processes again.
• Marking a partial failure "failed": a retry charges the customer twice.

Day 3Circuit Breakers

Fail fast when it's clearly broken. Timeouts and retries still wait 3.5 s per call and pile load onto a failing bank. A circuit breaker stops calling: users get an error in ~10 ms and the bank can recover.

Closed, open, half-open. Closed: calls pass and failures are counted. Open: calls fail immediately until a timer expires (often 30–60 s). Half-open: one test call decides, closed on success or open again.

Count what's really a failure. Timeouts, 5xx and refused connections count. 4xx errors and business rejections like "card declined" don't: the service worked correctly and said no.

Use failure rate, not count. "5 failures in 60 s" trips on Black Friday at 10,000 req/s even though 0.05% is normal. Open at a failure rate (around 50%) with a minimum number of calls.

Retry, breaker and bulkhead together. Check the circuit breaker first (cheapest), take a bulkhead slot second, retry transient failures last, with exponential backoff and jitter. They're layers, not alternatives.

• Closing to full traffic at once: the backed-up herd crashes the recovering service. Ramp up.
• Alerting only on error rate: an open circuit lowers it and hides the outage.
• A fallback for payments, auth or inventory: you can't fake those.

Day 4Webhook Delivery System

At-least-once plus idempotent receivers. True exactly-once is impossible (the Two Generals problem). Retry until acknowledged and give every event an ID the receiver uses as its idempotency key; a duplicate costs one lookup.

The pipeline. Event sources publish to a durable bus (Kafka); a dispatcher fans each event out to its subscriptions; delivery queues partitioned by subscription feed workers; exhausted retries go to a dead letter queue.

Retry what can succeed. Retry network errors, timeouts, 5xx and 429 (honour Retry-After). A 4xx means the request is malformed: retrying won't help, so log it or dead-letter it.

Sign and timestamp every webhook. HMAC-SHA256 over the exact bytes sent, with a timestamp inside the signature. Receivers compare in constant time and reject anything more than 5 minutes old, which stops replays.

Recover fast after an outage. Monitor endpoint health. When a dead endpoint recovers, move its pending retries to an immediate queue and ramp up gradually, within its rate limit, instead of waiting out the backoff.

• Re-serializing the payload after signing, so the receiver's HMAC over the raw body fails.
• Retrying 4xx responses that will never succeed.
• Delivering events for one entity in parallel: partition by entity to keep order.

Day 5Distributed Cron & Job Scheduling

Locks alone don't work. A plain lock stays held when its server crashes. A lock timeout lets a slow job run twice. A Redis lock released with GET then DELETE can delete the next holder's lock.

Perfect is impossible, so be practical. FLP says consensus can't be guaranteed with even one faulty process. Production cron runs at-least-once, makes jobs idempotent, tolerates small delays and alerts on missed runs.

One leader schedules. Elect a leader with a lease in etcd, ZooKeeper or Consul; followers wait to take over. Use a coordination service instead of building your own unless you build databases.

Fencing tokens stop stale leaders. Each new leader gets a higher token and sends it with every operation; resources reject older tokens. A deposed leader paused by GC can't do damage when it wakes up.

Track jobs outside the leader. Record executions in a job store with heartbeats. A new leader skips jobs still beating, a stuck-job detector reschedules silent ones, and checkpoints let a job resume instead of redoing work.

• A lock with no expiry: one crash and the job never runs again.
• Releasing a lock you may no longer own; check and delete atomically.
• Non-idempotent jobs: a failover rerun sends the same 50 emails again.

BossThe Ultimate System Design Interview

The spike is the problem. 5 million users hit one 10-minute sale and 80% of traffic lands in the first 30 seconds: ~150,000 requests a second at peak. CDN, WAF and rate limits absorb the herd before it reaches services.

Atomic claims in Redis. A Lua script checks the user's existing claim and decrements the inventory counter in one atomic step, restoring it if it goes below zero. No database in the hot path, so no overselling.

Claims expire back into the pool. A claim holds a unit for 3 minutes. A claim-expiry worker returns unpaid units to inventory, triggering a second wave of claims (~15,000 claim operations for 10,000 units).

Checkout uses every Week 2 pattern. Validate the claim, check the idempotency store by claim_id, call Stripe behind a circuit breaker with a timeout and its idempotency key, create the order in a transaction, publish events asynchronously.

Precompute notifications. Build the 2.4M-recipient list 10 minutes before the sale into Redis batches. At start, 24 workers fan out at once and queue everything in ~10 seconds; push and email have separate pools.

• Checking stock then decrementing in two steps: a race that oversells.
• Computing millions of notification recipients at the moment the sale starts.
• Putting the database in the claim path at 40,000 claim attempts a second.

Day 1Queue vs Stream

Work to do vs things that happened. A queue holds jobs: each message goes to one worker and is deleted after the ack. A stream is an append-only log: events stay, each reader tracks its own offset and can replay.

Consumer groups. Kafka's model: every group gets all messages (fan-out) and within a group each partition goes to exactly one consumer (load balancing). Parallelism equals the partition count.

Order lives in the partition. Kafka orders messages only within a partition. Key by the entity that needs order (order_id, user_id) so "created, paid, shipped" can't be processed out of order.

At-least-once plus idempotent consumers. Exactly-once needs special support on both sides, so most systems ack after processing and dedupe, e.g. by partition and offset stored in the same transaction as the work.

• Kafka for simple background jobs that a queue with built-in retry would handle.
• No partition key, so one order's events land on different partitions.
• One partition "to start": only one consumer can ever work on it.

Day 2Transactional Outbox

The dual-write problem. Saving an order then publishing to Kafka are two writes with no shared transaction. A crash in between leaves an order no downstream service hears about; flipping them publishes events for orders that never saved.

Write the event to the database. Insert the business row and an outbox row in the same transaction: both commit or neither does. A separate publisher reads unpublished outbox rows and sends them to Kafka.

Polling or CDC. A poller queries published_at IS NULL every few seconds: simple, but adds latency and database load. CDC (Debezium) tails the write-ahead log: ~10–100 ms latency, no polling, more to operate.

Still at-least-once. A publisher that crashes after Kafka acks but before marking the row published sends it again. Consumers dedupe by event ID; Kafka transactions can't cover the database update.

• Writing the outbox row in a separate transaction: the dual write is back.
• Publishing to Kafka inside the request path, so Kafka trouble fails user requests.
• No alert on outbox lag: the table grows to 10M rows before anyone notices.

Day 3Backpressure and Flow Control

Async removes natural backpressure. A synchronous client can't send faster than the server answers. A queue absorbs the gap instead, until memory runs out: absorption, then resource exhaustion, then collapse.

Watch lag, depth and latency. Consumer lag (alert at 10,000 messages or 5 minutes), queue depth (100,000 or growing for 5 minutes) and queue wait time. Throughput falling while lag grows means consumers are struggling.

Four responses. Shed low-priority load (metrics, logs), rate-limit producers (adaptively, as lag grows), scale consumers (Kafka: up to the partition count), or spill overflow to disk or S3 for later.

You get two of three. Completeness, latency, stability: pick two. Most systems keep completeness and stability and accept higher latency during spikes; real-time pricing drops stale events instead.

Recover gradually. After an outage, don't jump from 100 to 10,000 messages a second: the backlog floods downstream. Step up (500, 1,000, 2,500, 5,000, 10,000) after a minute of stability each.

• Unbounded in-memory queues: the OOM killer takes every message with it.
• Rate limiting only at the API gateway while internal services overwhelm the database.
• Hot partition keys: one consumer drowns while the others idle.

Day 4Dead Letters and Poison Pills

A poison pill blocks everything behind it. A message that crashes the consumer every time causes a crash loop: restart, fetch the same message, crash. In the lesson, one bad message held 50,000 orders for 4 hours.

Classify the failure. Transient (timeouts, rate limits): retry with backoff. Permanent (bad JSON, missing field, rule violation): straight to the DLQ. Poison pills (crashes, hangs, memory blowups): quarantine immediately.

Delayed retry topics. Instead of retrying in place, republish to retry topics with growing delays (1 s, 10 s, 60 s), then the DLQ. The main topic keeps flowing; Uber described this design for Kafka.

A DLQ is an operational tool. Store the original message, error and stack trace, retry count, timestamps and source topic, partition and offset. Monitor depth, growth and age, triage daily, and build replay tooling from day one.

Replay needs idempotency. Check the payment_id before reprocessing so a replay can't double-charge. Old messages may not match today's schema: version them, migrate on replay, or close them out.

• Resetting the retry count to 0 in each consumer instead of reading it from headers: infinite loop.
• A DLQ nobody monitors: it becomes a graveyard of lost data.
• A DLQ consumer that processes instead of just storing and alerting.

Day 5Audit Log System

Who did what, to which resource, when, from where. An audit log serves auditors, legal and security, not engineers: strict schema, retention in years, and it must be immutable. An application log is for debugging and can be deleted.

Capture in the same transaction. Write the audit event to an outbox in the same transaction as the business action, so a transfer can't succeed without its record. Kafka, consumers and the DLQ carry it asynchronously.

Immutable and tamper-evident. No UPDATE or DELETE: corrections are new events. Each event carries the previous event's hash, checkpointed somewhere an attacker can't change, and cold files sit under S3 Object Lock.

Tiered storage. Hot (0–90 days) in TimescaleDB for fast investigation queries, warm (to 1 year) in ClickHouse or BigQuery, cold (1–7 years) as Parquet on S3, queryable with Athena.

Never shed audit events. When consumers fall behind, scale them, batch harder and alert, but accept latency: audit data can't be dropped. Sort investigations by ingestion time and keep both timestamps.

• Writing the audit event after the business transaction: money moves with no trail.
• "Fixing" audit rows with UPDATE, which breaks the chain of custody.
• Logging only "delete doc_123" without actor, context or reason.

BossEvent-Driven Food Delivery Order System

Size the dinner rush. 500,000 orders a day, 60% in 6–9 PM: ~28 orders/s, ~56 at the peak, design for 100. With ~8 events per order and 7 consumer services, that's 5,600 event deliveries a second.

Kafka keyed by order_id. Seven services need the same events, replay helps debugging, and each order's events must stay in order. Partition orders.events (32 partitions) by order_id; each service is its own consumer group.

Different consumers, different guarantees. Payment needs effectively-once with an idempotency key per order (Stripe won't charge twice). Analytics can lag, push is fire-and-forget, fraud detection needs real time.

Retry topics, then the DLQ. Transient failures go to retry topics delayed 1, 5 and 15 minutes; after 3 retries, the DLQ. Payment failures retry then notify the customer; restaurant rejections find an alternative or refund.

Audit for disputes. 2% of orders are disputed. Every service writes immutable audit events, so support queries one order's full timeline in minutes instead of correlating logs across services.

• Saving the order and publishing to Kafka as two writes: lost or phantom orders.
• Shedding driver-assignment events: a late driver hurts more than late analytics.
• Retrying forever in the main topic, so one failing order blocks the rest.

Day 1Caching Patterns

Why cache. Latency (database 10–100 ms, Redis 0.5–2 ms, local memory 0.0001 ms), throughput (5,000 database queries/s vs 100,000+ Redis ops/s) and cost. "Just add Redis" isn't a strategy.

Cache-aside is the default. The app checks the cache, on a miss reads the database and fills the cache, and on writes invalidates the key. Only used data gets cached, and a cache outage just makes things slower.

Write-through vs write-behind. Write-through updates cache and database together: fresh reads, slower writes, for balances and inventory. Write-behind acks after the cache and flushes later in batches: fastest, but a cache crash loses data.

Mix patterns per data type. In one store: catalog cache-aside, inventory write-through (or no cache during a sale), prices cache-aside with event-driven invalidation, view counts write-behind. Checkout always reads the database.

TTL is the safety net. Invalidation can fail on a Redis timeout. A TTL sized to the staleness you can accept means a stale value still expires: product details 1 hour, prices 5 minutes, inventory 30 seconds.

• Letting a cache error become a 500 instead of falling back to the database.
• Caching everything: rarely read data evicts the hot set and lowers the hit ratio.
• Collision-prone keys like user; use product:{id}:v1 with a schema version.

Day 2Invalidation Strategies

The cache doesn't know the data changed. A flash sale moved a price from $99 to $79, but the 1-hour TTL kept showing $99 and 147 customers complained. Match each data type's invalidation to its staleness tolerance.

TTL, events, versions. TTL: simple, stale until expiry. Event-driven: publish on change and delete the key in ~100–500 ms, but needs Kafka and a DLQ. Versioned keys: bump product:123:v5 to v6, or a global version to invalidate everything.

Delete, don't update. Write-update races: two writers can leave the cache holding the older value. Deleting is idempotent and the next read loads fresh data; update in place only with one writer or atomic ops like INCR.

Invalidate after commit. Deleting inside the transaction lets a reader cache the old row before commit, leaving cache and database apart indefinitely. Facebook's leases go further: a fill after an invalidation is refused.

Decompose the cache. Cache product:123:details (24 h), price (5 min plus events), inventory (30 s plus events) and reviews (1 h) separately, so an inventory change doesn't evict the description.

• Invalidating only product:123 and forgetting category pages, search results and deals.
• Pattern deletes (KEYS category:*) in the hot path; track keys per category in a set.
• Event-driven invalidation with no TTL: one lost event means stale forever.

Day 3Thundering Herd

One expiring key can take the site down. When a hot key expires, every concurrent request misses and queries the database: 1 miss becomes N queries. 50,000 homepage requests met a database built for 5,000 queries a second.

Coalesce misses into one fetch. Request coalescing (single flight) lets concurrent requests for a missing key join one in-flight fetch and share its result. Instagram caches a promise for the value; locking does the same with more waiting.

Never let hot keys expire. Background refresh recomputes known hot keys (homepage, trending, deals) before their TTL, so users never miss. For a scheduled change, pre-warm the new key and switch atomically.

Serve stale while revalidating. When an entry goes stale, return it immediately and refresh in the background. If the refresh fails, extend the stale window instead of letting it lapse onto a struggling database.

Desynchronize expiry. Jitter TTLs and refresh probabilistically just before expiry, weighted by how long recomputing takes, so keys loaded together don't all expire together.

• One TTL for everything, so the homepage expiring sends 10K queries at once.
• Coalesced waiters with no timeout; fall back to stale data instead.
• Deploying with a cold cache; warm hot keys behind a health-check gate first.

Day 4Feed Caching

Reads vs writes pull apart. Computing a feed on read means querying every followee (500 × 10 ms = 5 s). Precomputing it means every post fans out to every follower: one celebrity post is 50 million cache writes.

Fan-out on write. When a user posts, push the post ID into each follower's feed cache (a Redis sorted set scored by time). Reads become one cache lookup; the cost moves to writes and storage.

Hybrid push and pull. Push posts from regular users (under ~10K followers); store celebrity posts once and pull them at read time, merging with the precomputed feed. Twitter and Instagram run mostly push.

Cache IDs, trim, skip the dormant. Store post IDs, not posts, and trim each feed (~800 entries). Skip long-inactive followers during fan-out and rebuild their feed when they return; fetch pulled posts in parallel.

• Pure push for everyone: one celebrity post backs up the fan-out for an hour.
• Feeds never trimmed, so sorted sets grow forever.
• Fetching each followee's posts sequentially instead of in parallel.

Day 5Multi-Tier Caching

Five layers, five jobs. Browser (static assets, preferences), CDN (public content, media), API gateway (auth validation, rate limits, dedup), Redis (business data, sessions, flags), database buffer pool (automatic).

Invalidate closest to the database first. Delete the Redis key, then the gateway cache, then purge the CDN asynchronously. Purge the CDN first and it may refetch the old value from a still-stale origin cache. Browsers expire only by TTL.

Never let the CDN cache personal data. Mark authenticated responses Cache-Control: private. Vary on headers that change the content (Accept-Language, Accept-Encoding), never on Cookie: that gives every user their own cache entry.

Bounded staleness, not perfect consistency. Design a maximum staleness per tier: CDN 60 s, Redis 5 minutes (usually seconds with events), browser 60 s for API data. Checkout bypasses every cache.

• An authenticated /api/profile cached at the CDN: one user's data served to another.
• Missing Vary: Accept-Language, so Spanish users get the English page.
• Unversioned keys: a deploy reads the old schema; put v3 in the key.

BossDesign a Global E-Commerce Product Catalog System

At 600K req/s, the hit rate is the design. 500M daily views is ~6,000 req/s, and flash sales spike it 100x. Even 1% misses at 600,000 req/s means 6,000 database queries a second: hit rate must exceed 99%.

Freshness per data type. Metadata tolerates an hour (TTL), prices under 5 seconds (events plus a 30 s TTL), inventory under 30 seconds for display, and checkout reads live data. Images can be a week old.

Flash sale protection. Pre-warm the deal before midnight, coalesce concurrent misses into one fetch, refresh hot keys in the background and serve stale while revalidating. Last Black Friday a single expiring key took the site down.

Personalization by activity tier. Precompute recently viewed and recommendations for active users, give recent users shorter TTLs, compute on demand for dormant ones. Caching all 100M users' 3.5 KB would waste most of 350 GB.

Degrade when Redis fails. A circuit breaker detects the failure in under a second; serve stale data from a local in-process cache, hide personalization, pause flash-sale purchases to avoid overselling, then recover half-open after 30 seconds.

• Letting the Deal of the Day key expire at midnight instead of pre-warming it.
• Caching prices at the CDN: checkout then disagrees with the page.
• Precomputing personalization for every registered user, active or not.

Day 1Consistency Models

Who sees a write, and when? A write lands on Node 1 and a read hits Node 2 before replication arrives. Consistency models are precise answers to what that read may return: new value, old value, or an error.

The spectrum, strongest first. Linearizable (behaves like one copy), sequential (one agreed order), causal (cause before effect), read-your-writes, monotonic reads (never go backwards), eventual (converges, no promise when).

Session guarantees users notice. Read-your-writes: route a user's reads to the primary after their write, or track the write version. Monotonic reads: remember the last version read and only accept replicas at or past it.

Choose per operation. Browse products eventually consistent, reserve inventory and check out strongly consistent, order history read-your-writes. DynamoDB and Cassandra let you choose per read: eventual, QUORUM, ALL.

Spanner buys strong consistency with time. TrueTime (GPS and atomic clocks, uncertainty typically under 7 ms) plus Paxos and commit-wait give global linearizability, at ~10–20 ms per write and 100 ms+ across regions.

• Reading from a replica right after writing to the primary: the user's change "disappears".
• Random replica per read, so a counter appears to go backwards.
• Strong consistency for everything: latency and availability pay for reads that don't need it.

Day 2Distributed Transactions — The Saga Pattern

Why not two-phase commit. 2PC holds locks from PREPARE until the decision, blocks if the coordinator dies, needs every service up at once, and most NoSQL stores, brokers like Kafka and third-party APIs (Stripe) can't take part.

A saga is local steps plus compensations. Each step is a local transaction in one service; if step N fails, run compensations for steps 1 to N-1. No distributed locks, no coordinator holding them.

Compensation is not rollback. A refund is a new transaction that reverses the charge; both stay in history. Compensations must be idempotent, check state first (no-op if the step never ran) and eventually succeed.

The pivot transaction. The point of no return, like shipping. Before it, recover backward (compensate); after it, recover forward (retry until done). In the Airbnb example the pivot is capturing payment.

Choreography vs orchestration. Choreography: services react to each other's events; fine for 2–4 steps but the flow is hard to see. Orchestration: a coordinator calls each step and persists saga state; better for 5+ steps and branching.

• A refund compensation that can fail with no retry: the saga sticks with the customer charged.
• Payment steps without an idempotency key like saga-{id}-payment: a timeout retry charges twice.
• Compensating everything blindly instead of tracking which compensations finished.

Day 3Saga Orchestration Deep Dive

Durable execution. Temporal records every activity result in an event history. After a crash the workflow replays from the start, completed activities return their stored results instead of running again, and execution continues where it stopped.

Workflows decide, activities act. Workflow code must be deterministic because it replays: no random numbers, clock reads, UUIDs or HTTP calls (use workflow.now(), workflow.random(), activities). Activities do the side effects, with timeouts and retries.

Signals and queries. A signal delivers an external event (manager approved, rider cancelled) to a waiting workflow, which can sleep for days at no cost. A query reads workflow state without blocking it.

Version and continue-as-new. workflow.patched("fraud-check-v2") keeps in-flight workflows on the old path while new ones get the change. Long-lived workflows (subscriptions) use continue-as-new so history stays bounded.

• Calling datetime.now() or an HTTP API inside workflow code: replay diverges.
• A workflow that loops forever without continue-as-new: unbounded history.
• Activities without timeouts, so one hung call stalls the workflow.

Day 4Conflict Resolution

Last-write-wins loses data. The highest timestamp wins, so of two concurrent writes one silently disappears, and clock skew can drop the truly latest one. Fine for a location, a profile picture or "last seen"; wrong for carts and counters.

Vector clocks detect concurrency. Each node keeps a counter per node, increments its own on every write and takes the max on receive. If neither clock is ≤ the other in every entry, the writes were concurrent: a real conflict.

CRDTs merge by construction. Merge is commutative, associative and idempotent, so replicas converge whatever the order. G-Counter sums per-node counts; PN-Counter pairs two; OR-Set tags each add so a concurrent add beats a remove.

Pick by data semantics. Latest is right: LWW. Need to detect and ask: vector clocks with siblings (Riak allow_mult). Must merge offline: CRDTs. Collaborative text: OT (Google Docs) or a sequence CRDT (Yjs, Automerge).

• A counter as an LWW value: concurrent increments vanish. Use a PN-Counter.
• A plain set for add and remove: concurrent operations resurrect or drop items. Use an OR-Set.
• Never garbage-collecting tombstones, so CRDT state grows forever.

Day 5Leader Election and Coordination

Timeouts can't tell slow from dead. A GC pause, slow network or clock skew makes a live leader look dead; a follower takes over and two leaders accept writes: split-brain. You can't guarantee both safety and liveness, so favour safety.

Fencing tokens stop zombie leaders. Every election issues a higher token. The leader sends it with each request and resources reject anything below the highest they've seen, so a leader waking from a pause can't corrupt data.

Raft in one breath. Followers expect heartbeats; on an election timeout one becomes a candidate, bumps the term and asks for votes; a majority makes it leader. At most one leader per term, and terms act as fencing tokens.

Don't build it: ZooKeeper or etcd. ZooKeeper's ephemeral sequential znodes make the lowest sequence number the leader, deleted when its session expires; watches announce changes. etcd gives leases on Raft.

Epochs everywhere. Kafka elects partition leaders from the in-sync replicas and bumps an epoch so an old leader is rejected; Redis Sentinel needs a quorum to declare the master down; Kubernetes controllers hold a Lease object.

• A lock with no fencing token: a paused holder wakes up and writes over the new holder.
• "No heartbeat in 5 s, so I'm leader" without asking a quorum.
• Leases on wall-clock time; use monotonic clocks and renew early.

BossMultiplayer Game Matchmaking System

Size it with Little's law. 2M concurrent players, 1.6M in 5v5 matches (160,000 matches of 20 minutes): 8,000 matches start a minute, ~1,300 queue joins a second. At a 30 s wait, only ~40,000 players sit in queue.

One matchmaker per region, fenced. Two matchmakers reading the same queue would put the same 10 players in two matches. Elect one leader per region in etcd, and make every queue change carry its fencing token, checked atomically in Redis.

Ratings need strong consistency. Write ratings to the primary; a player viewing their own rating gets read-your-writes; the queue snapshots the rating from the primary at join time; leaderboards read cached replicas.

Match creation is a saga. Remove players from the queue, ready check, allocate a server, write the match, notify. If allocation fails, compensate by re-queuing confirmed players with priority; a leader crash leaves pending_match_id records to clean up.

Party state as an OR-Set. Kicks, leaves and invites can happen concurrently on different devices. With an OR-Set the merge is deterministic: concurrent removes of Carol both take effect, Dave's add survives. Parties match on their highest-rated player.

• Two matchmakers without fencing: players double-booked into two games.
• Matching on a stale replica's rating right after a match, creating unfair games.
• Removing players from the queue with no compensation if server allocation fails.

Day 1Notification Platform — Problem Understanding & High-Level Design

Ask before you draw. Spend the first five minutes on questions (FERNS: functional, edge cases, resources, non-functional, scale). Here they reveal five notification types, five channels, preferences, quiet hours, digests, fallback and compliance.

Priority decides everything. Critical security alerts go out on several channels at once with maximum retries; transactions are high priority with a fallback; marketing is low, batchable and delayable. Marketing must never delay transactions.

Every channel behaves differently. Push is free but best-effort, with tokens that rotate. Email is cheap, slow and fights spam filters. SMS is reliable but expensive with a 160-character limit. In-app needs a notification center for offline users.

The pipeline. Send API writes the notification and an outbox row in one transaction; the outbox publishes to priority Kafka topics; a router applies preferences and quiet hours; channel workers render and call providers.

Compliance is a requirement. CAN-SPAM needs a working unsubscribe and opt-outs within 10 business days; GDPR needs explicit marketing consent; TCPA bars marketing SMS outside 8am–9pm local and honours STOP. Keep a full audit trail.

• Drawing boxes before clarifying scale, latency and compliance.
• Sending marketing through the same queue as security alerts.
• Using SMS as a default channel: push first, email second, SMS last.

Day 2Core Notification Flow

The Send API returns fast. Validate, assign a notification ID, save it with an outbox row in one transaction and return the ID. Delivery happens asynchronously, idempotency keys make retries safe, and bad requests never reach the queue.

Why the outbox here. Save-then-publish can save a notification that's never processed (the user is never told) or process one that was never saved (no status, possible duplicates). The outbox makes the save and the event atomic.

Route by preference. The router consumes from Kafka, reads the user's preferences (Redis first, then PostgreSQL), checks quiet hours and channel availability, and publishes one task per channel topic.

One interface for every provider. FCM, APNs, SendGrid and Twilio sit behind the same provider interface. Channel workers consume their own topic, retry with exponential backoff and send exhausted messages to a dead letter queue.

Status is a state machine. PENDING, QUEUED, ROUTING, SENDING, then SENT, FAILED or BOUNCED; SENT can still become DELIVERED or BOUNCED when provider webhooks arrive later, then READ. FAILED goes back to QUEUED for a retry.

• Waiting for delivery inside the Send API instead of returning the ID.
• Letting invalid payloads into the queue, where they fail late.
• Treating a provider's "accepted" response as delivered.

Day 3Advanced Features

Batch what would be noise. 50 trades, 20 likes or 10 transactions shouldn't be 50, 20 or 10 notifications. Batch by count, by time window or both ("20 people liked your post") with a worker that flushes batches.

Digests and scheduling. Digests are timezone-aware daily, weekly or monthly summaries configured per user. Scheduled and recurring (cron) notifications convert to the user's timezone, can be cancelled, and the scheduler runs only on the elected leader.

A/B test templates. Templates carry variants with weights. Assign each user consistently to one variant, measure engagement and check statistical significance before declaring a winner.

Multi-channel delivery is a saga. Critical alerts go out on primary channels together, track each channel's state, fall back on failure and escalate: push first, SMS if unopened after 5 minutes, a phone call at the 15-minute check.

The in-app notification center. Store notifications with read/unread state and grouping, push them over a WebSocket gateway when the user is online. Preferences are hierarchical: global, then category, then channel, with quiet hours.

• Sending every like or trade as its own push notification.
• Scheduling in server time instead of each user's timezone.
• Updating preferences without a version check: optimistic locking prevents lost updates.

Day 4Scale, Reliability & Edge Cases

Campaigns arrive as a wall. 10M in an hour is only 2,800/s on top of ~5,800/s, but marketing clicks Send once: 100,000+/s for the first seconds. Rate-limit campaign ingestion, batch it and isolate it on low-priority topics and workers.

Hot users fan out. A celebrity post to 1M followers means 1M preference and token lookups at once on a hot partition. Fan out asynchronously in batches, query followers in batches and route partition-aware.

Respect provider limits, survive provider failure. Per-provider sliding-window rate limiters with backoff on rate-limit errors; a circuit breaker per provider; automatic failover (SendGrid to SES, Twilio to SNS); degraded modes that shed load.

The edge case encyclopedia. Rotating and stale device tokens, multiple devices, hard vs soft email bounces and spam complaints, quiet hours across DST, and users who change preferences or are deleted mid-delivery.

Re-check at delivery time. A user changed their email while a notification was queued for the old one. Read preferences and contact details again when delivering, not only when queueing.

• Letting a campaign burst share workers with transaction notifications.
• Generating 1M notification requests synchronously for one celebrity post.
• Keeping stale device tokens and bounced addresses: they waste sends and hurt reputation.

Day 5Operations, Monitoring & Interview Mastery

Three pillars of observability. Metrics for what's happening now (delivery rate, queue depth, provider latency, circuit states), structured logs, and distributed traces that follow one notification from API to router, worker, provider and callback.

Alerts that don't cry wolf. Page when delivery drops below 95%, warn when a circuit opens, investigate when queue lag passes 1M. Every alert has a runbook: low delivery rate, Kafka backlog, provider circuit open.

Separate ingestion from delivery. API, Kafka, router, channel workers, providers, then callbacks. Ingestion returns immediately; Kafka buffers and enables retry; delivery scales and fails on its own.

The 45-minute framework. Requirements 0–5 min, estimation 5–10, high-level design 10–25, two or three deep dives 25–40, wrap-up 40–45. Cover multi-channel delivery, reliability, scale, preferences and observability.

Lessons from the giants. Uber: P0/P1/P2 priority classes on separate infrastructure. Slack: online users get WebSocket, offline get push after a delay. WhatsApp: push only says "check for messages", content stays end-to-end encrypted.

• No traces, so a slow notification can't be pinned to the router, worker or provider.
• Alerts without runbooks, or so many that on-call ignores them.
• Committing Kafka offsets before the provider call succeeds.

BossFood Delivery Order System

Size the dinner rush. 5M orders a day is ~58 a second; dinner is 4x, ~230 a second. Each order brings status updates, location points and notifications: ~7,000 writes and ~60,000 reads a second at peak.

Never lose an order. Write the order and an outbox row in one transaction; the publisher marks rows published only after Kafka confirms, so a crash just republishes, and idempotent consumers absorb the duplicate.

Payments: authorize, capture, refund. Authorize at order time, capture when the driver picks up, refund on cancellation, every Stripe call with an idempotency key. Without one, a retried authorization after a timeout creates a second PaymentIntent.

No driver assigned twice. Two orders can both see Driver X as free and both take them. Assign atomically with row locking and a version check, so only one assignment succeeds and the other picks another driver.

The order saga. Restaurant rejects after payment is authorized: void the authorization and notify the customer. Payment capture fails after pickup: the food is gone, so retry with backoff, then flag for manual review without blocking delivery.

• Saving the order, then publishing to Kafka: Kafka down means a charged customer and no food.
• Retrying payment calls without an idempotency key.
• Blocking delivery because payment capture failed after pickup.

Day 1Search Fundamentals & Architecture

Why LIKE fails. LIKE '%red%' can't use a B-tree, which is sorted by string start, so 50M rows get a full scan. It also has no relevance ranking, typo tolerance, word boundaries, stemming or synonyms.

The inverted index. Pre-compute word to documents instead of scanning documents for words. A query is one lookup per term plus an intersection, smallest list first: "running shoes" takes under a millisecond.

Text analysis. Character filters (strip HTML), then the tokenizer, then token filters (lowercase, stop words, stemming, synonyms). The same analyzer must run at index time and query time, or "Running" never matches the indexed "run".

The multi-field pattern. One source field, several representations: brand as text for search, brand.keyword for facets and sorting, brand.autocomplete for typeahead. IDs are keyword: they're only exact-matched.

Scatter-gather. A coordinating node sends the query to every shard (primary or replica). Each searches its own inverted index and returns its top N with scores; the coordinator merges, sorts by score and returns the overall top N.

The product search cluster. 20 primary shards plus 20 replicas on 6 data nodes, 3 dedicated master nodes, 2 coordinating nodes. Text goes in must to drive the score; category, price and stock go in filter.

• Different analyzers at index and query time, so tokens never match.
• Oversharding: per-shard memory and file-handle overhead degrades queries.
• Not planning primary shards: the count is fixed at creation, so changing it needs _split/_shrink or a reindex.

Day 2Indexing Pipeline

Never dual-write. Writing PostgreSQL then Elasticsearch breaks four ways: the ES write fails, the process crashes between writes, a partition holds the transaction open, or retries reorder updates so ES ends at $100 while PostgreSQL has $120.

Change data capture. App to PostgreSQL; Debezium reads the WAL through logical replication and publishes change events to Kafka; the indexing pipeline consumes them into Elasticsearch. The app writes to one place, and events replay within Kafka's retention.

Version by WAL position. Each bulk operation carries version: event.lsn with version_type: external. If an older change arrives late after retries or reordered batches, ES rejects it with a version conflict instead of overwriting newer data.

Batch, but by priority. Flush at 500 documents or after 5 seconds, so quiet periods don't strand updates. Stock going to zero is CRITICAL in 10-document batches; description edits are NORMAL at 500; bulk imports LOW at 1,000.

Zero-downtime reindex. Reads stay on the products alias over v1 while updates go to both indexes and old data is copied with op_type: create. Verify counts, switch the alias in one atomic remove+add, then delete v1.

• Running a big import through the live pipeline, starving CDC updates.
• Skipping index aliases, so a reindex can't switch with zero downtime.
• Blaming the pipeline when the ES document is right but a result cache is stale.

Day 3Query Processing & Relevance

BM25 in one line. Score grows with term frequency but saturates (k1, default 1.2), rare terms weigh more (IDF), and long documents are normalized (b, default 0.75). 10 occurrences aren't 10x better than 1.

Rare terms win. In the example corpus, "nike" (2% of documents) has IDF 3.912, "shoes" (10%) 2.3026, and "the" (90%) 0.1054. A match on the rarer term says more about relevance.

Query context vs filter context. What the user typed is scored in query context. What they selected (facets, toggles, ranges) goes in filter: yes or no, unscored, cacheable as a bitset. The lesson puts the split at ~3x faster.

Boost in layers. Start with field boosts (name^3, brand^2, description), add business boosts in should clauses, and use function_score (popularity, recency decay, rating, featured) only when needed. A/B test every ranking change.

Never return nothing. Progressive relaxation: exact AND query, then drop low-priority filters (rating, price), then fuzzy matching, then any-term matching, then no filters at all. Category stays: it's a strong intent signal.

Pick the multi_match type. best_fields with tie_breaker 0.3 is the product default. cross_fields lets "John" match first_name and "Smith" last_name. most_fields sums scores across analyses. phrase_prefix and bool_prefix serve autocomplete.

• Putting category, brand and price in must: they sway scores without adding relevance and aren't cached.
• Filtering out-of-stock items away when they should be demoted with a boosting query.
• Over-boosting, which makes relevance unpredictable.

Day 4Advanced Search Features

Edge n-gram autocomplete. Pre-compute prefixes at index time: "nike" is stored as "ni", "nik", "nike", so each keystroke is an exact token lookup. The search analyzer only lowercases, or "nik" would also match every "ni" word.

"Did you mean". A phrase suggester over word shingles, with collate and prune so only suggestions that find documents survive. Show it only when the correction has over 2x the original's results and more than 10.

Multi-select facets. User selections go in post_filter, which narrows hits but not aggregations. Each facet counts as if its own filter weren't applied: tick Nike and the brand facet still shows Adidas.

Synonyms, both kinds. Two-way groups (tv, television, telly) and one-way rules (ipod => ipod, mp3 player), so "mp3 player" isn't pulled to iPods. Query-time synonym_graph handles multi-word; updateable search analyzers reload without a reindex.

A field per language. One index with name_es, name_fr and so on, each with its own analyzer (spanish, kuromoji), because stemming, stop words and tokenization differ. Separate indexes separate cleanly but make cross-language search harder.

Personalize with boosts, not filters. Brand and category affinities from 30 days of behavior (purchases weigh 3, views 1) become function weights scaled by personalization_strength. Recent views get 0.8. Other brands still show, avoiding a filter bubble.

• N-gramming the query as well, so "nik" also matches every "ni" word.
• Letting a facet's own selection narrow its counts, hiding the other brands.
• Making brand synonyms two-way, pulling "mp3 player" searches to iPods.

Day 5Search Operations & Scale

Dedicated node roles. 3 dedicated master nodes give a quorum that survives one loss without split brain. Coordinating nodes route and aggregate, shielding data nodes; data nodes are memory-optimized with SSDs.

Size shards for growth. Primary count can't change later, so plan ahead: 620 GB of index × 1.5 growth is 930 GB; at 30 GB per shard that's ~31, rounded to 32 primaries, 64 shards with one replica.

The largest requirement wins. Count data nodes separately for storage, memory and CPU, take the largest (minimum 3) and round to even for AZ spread. At 50K QPS about 330 cores make CPU the limit: 6 data nodes become 22.

Pre-scale, don't reactive scale. New nodes need time to join and take over shards. Scale up 24 hours before Black Friday with an extra replica, peak at 22 data and 10 coordinating nodes, then scale down gradually.

Degrade gracefully. Warning (p99 > 500 ms or CPU > 70%): longer cache TTL, no spell check or personalization. Critical (p99 > 1 s or CPU > 85%): also no facets or highlighting, simpler queries. Better slow search than no search.

Snapshots plus CDC. Lose the cluster: provision a new one, restore the latest S3 snapshot, resume CDC so Kafka replays later changes. For corruption, restore beside the live index and swap the alias. Untested backups are not backups.

• Force-merging a live, still-written index to one segment: the huge segment piles up deleted docs.
• Aggregating on text fields, loading fielddata onto the heap; use keyword.
• Deep from offsets instead of search_after.

BossDesigning a Legal Document Search System

Size the corpus. 100M documents (50M public, 50M private) at ~65 KB is 6.5 TB of text. 2x index overhead and 2x for replicas make 26 TB in Elasticsearch, plus 50 TB of original PDFs in S3: ~80 TB.

Size the traffic. 50,000 concurrent users at 20 searches an hour is 1M an hour, ~280 a second; autocomplete at 5 keystrokes per search is ~1,400 a second. Today's 10,000 a minute (~170/s) leaves ~1.7x headroom.

A real query parser. A lexer and parser turn (breach /5 contract) AND damages NOT punitive into must, should and must_not clauses. Proximity becomes span_near with in_order: false; /s means ~15 words, /p ~50.

Isolation in layers. Per-tenant indices, a service-injected tenant_id filter, document-level security, a tenant taken from the validated JWT (never the request body), and audit logs. Zero tolerance means no single control is trusted alone.

Citations as edges. A separate index of {citing_id, cited_id, treatment} makes "who cites X" a term query on cited_id, with aggregations for treatment breakdowns, "good law" checks and 2-hop chains.

Triage every upload. Store the original in S3 first. A PDF with under 100 characters of text in its first pages is scanned: render each page at 300 DPI and OCR it. Word goes through Tika. Then publish to Kafka.

• Trusting one isolation control, or a tenant_id from the request body.
• Using a phrase query for proximity: "liability for negligence" should match negligence /10 liability.
• Fuzzy matching citation variants instead of a canonical case ID plus name synonyms.

Day 1Event Ingestion at Scale

Events are immutable facts. Each event has a unique event_id, an event time, a type and context. Never update one: record the change as a new event (event.processed) pointing to the original, or replay and the audit trail break.

Evolve schemas safely. A registry enforcing BACKWARD compatibility rejects a new required field like session_id, since old events lack it. Add fields as optional with a default. Name events domain.entity.action, in the past tense.

Collect, validate, then Kafka. Lightweight collectors validate size, required fields, schema, timestamps, business rules and security, then write to Kafka asynchronously. In a batch, valid events go to Kafka and invalid ones to the dead letter topic.

Timestamps and enrichment. An event time over 5 minutes in the future is an error; over 30 days old is only a warning, since offline phones send those. Enrichment (geo, user, product, session) is best-effort: a failed lookup never blocks.

Key by user, write idempotently. Kafka orders only within a partition, so user events are keyed by user_id. acks=all makes a write durable across in-sync replicas; enable_idempotence stops a retried send from writing a duplicate.

Shed telemetry, never revenue. Above 90% queue usage the collector sheds page.viewed-type telemetry but never orders, payments or sign-ups. Per-client limits (1,000 a second) and an adaptive limiter protect downstream systems.

• Events without an event_id: no deduplication, no tracing through the pipeline.
• Writing to the database before replying: a slow database makes a slow API.
• No dead letter queue, so a failed event is lost forever.

Day 2Streaming vs Batch Processing

Two ways to process. Batch processes large chunks: hours of latency, exact, cheaper, simpler. Streaming processes each event as it arrives: seconds of latency, approximate because late events may be missed, always on and more complex.

Lambda. Every event feeds a Spark batch layer (accurate, overwrites its views) and a Flink speed layer that fills the gap since the last batch; the serving layer adds them. Its big problem: two codebases that drift apart.

Kappa. One streaming codebase over a long-retention log. To reprocess, start a new version from the earliest offset, let it catch up, switch and delete the old output. 1B events at 100K/s replay in about 3 hours.

Watermarks. Watermark = max event time seen minus max out-of-orderness. With a 5-minute bound and latest event at 10:02 it's 9:57, so [9:00-10:00) fires only after an event at 10:05 or later. Wall-clock time plays no part.

Late data and state. An event after its window fired can be dropped, side-output, refire the window, or fit within allowed lateness. Keep large state in RocksDB with a TTL, and checkpoint so a crash doesn't start over.

Write once, run both. Apache Beam, Flink (batch is just bounded streaming) and Spark Structured Streaming run the same aggregation as a stream or a batch, so real-time and batch results can't drift apart.

• Writing batch output with mode("append"), so a retried job duplicates rows; overwrite the date partition.
• Keeping state forever with no TTL, until it runs out of memory.
• Grouping by a skewed key without salting, so one task runs for hours.

Day 3Data Modeling for Analytics

OLTP vs OLAP. OLTP runs the business: many small transactions, normalized (3NF), row storage. OLAP understands it: few large queries over many rows and few columns, denormalized, column storage. "Revenue by country" becomes one scan and a GROUP BY.

Star schema. A fact table of measures (revenue, quantity) at a stated grain, here one row per line item, with foreign keys to denormalized dimensions: product, customer, date, store. order_id is a degenerate dimension kept in the fact.

SCD Type 2. On a change, close the old row (valid_to, is_current = FALSE) and insert another. Facts store the surrogate key valid at sale time, so a plain join is point-in-time. Only tracked columns create versions.

Load dimensions, then facts. Dimensions first, so today's new versions have surrogate keys. Lookups use the version valid on the batch date, not is_current, so backfills stay right. A missing product maps to unknown key -1, keeping totals whole.

Columnar storage. SUM(revenue) over a 4-column table reads 1 column: 75% less I/O, better compression, vectorized. Parquet row groups carry statistics for predicate pushdown; Delta Lake and Iceberg add ACID transactions and time travel.

Partition and pre-aggregate. Partition by filter columns like date, Z-order by a second filter such as product_id, never partition by a high-cardinality key. A daily rollup by category scans 365 × ~100 = 36,500 rows instead of 1B.

• An order-level grain: you can never drill down to products.
• Business keys in the fact table instead of surrogate keys, so dimension history is lost.
• Partitioning by customer_id: 10 million partitions full of small files.

Day 4Late-Arriving Data and Correctness

Event time, not processing time. A purchase made at 14:30 on the subway reaches the server at 15:45, 75 minutes late. Windowed by processing time it lands in 15:00-16:00; by event time it lands in 14:00-15:00, where it happened.

Two separate knobs. Watermark = max event time minus the out-of-orderness bound: with a 5-minute bound and max 10:03 it's 9:58, and an older event can't lower it. Allowed lateness is separate: how long a fired window still accepts updates.

Per-key watermarks idle out. With per-key watermarks the global one is the minimum, so a phone offline for two days pins it in the past and windows stop closing. Keys not seen within the idle timeout are left out.

Tune lateness from data. Measure the lateness distribution (example: p50 2 seconds, p99 5 minutes, max 72 hours) and set it per source: minutes for web, hours for mobile. Unlimited lateness grows state until it runs out of memory.

Patch or reprocess. Patch additive metrics (SUM, COUNT) when late data is small. Reprocess from raw events, overwriting the range, for non-additive ones (AVG, MEDIAN, DISTINCT), big late volumes or changed logic.

Version, label, finalize. Store every computation as a version with a reason and delta, so latest, final and as-of queries work. Stream results are provisional; a daily batch job rereads raw events for final numbers. Label dashboards Live or Final.

• Windowing by processing time, so offline events land in the wrong hour.
• Overwriting results in place, leaving no answer to "why did yesterday change?"
• Alerting on live numbers; gate the normal threshold on is_final, a stricter one for live.

Day 5Query Layer and Optimization

Match the engine to the access pattern. Cloud warehouses (Snowflake, BigQuery) answer in seconds, managed, for ad-hoc SQL. Real-time OLAP (ClickHouse, Druid) gives milliseconds and high concurrency for user-facing dashboards. Trino queries the lake in place; DuckDB runs in-process.

Why the dashboard is slow. "Revenue by category, region and day for 90 days" scans ~2.7B of ~11B rows (30M a day), shuffles 50GB and takes 47 seconds: no partitioning, no pre-aggregation, no caching, joins that shuffle.

Partition, then cluster. Partitioning decides which files to read, and only helps when the query filters on the partition column. Clustering (sorting, Z-ordering) decides how rows sit inside files, so category = 'A' can skip files without it.

Pre-aggregate the predictable. Pre-aggregate patterns that are predictable and frequent (over 10 a day): the executive dashboard drops from 45 seconds to 50ms. Route to the coarsest rollup that keeps the needed granularity; leave rare ad-hoc slices on demand.

Cache with the right TTL. Layers: CDN, Redis query results (60-80% hits for popular queries), database cache, block cache. TTL follows freshness: about a minute for today, hours for history. Include the tenant in the key: same SQL, different rows.

Isolate tenants' data and compute. Row-level security is cheap, but one missing filter leaks data. Per-tenant quotas (10 concurrent queries, 100GB scanned a day, 300-second timeout) stop one noisy tenant from slowing everyone else.

• Caching everything for 24 hours, so live panels show stale numbers.
• One agg_everything rollup grouped by every dimension: nearly as big as the fact table.
• No query timeouts or byte limits, so runaway queries block others and costs explode.

BossReal-Time Analytics for Video Streaming Platform

Size the firehose. 10M peak viewers sending one event every 10 seconds is 1M events a second; a day adds up to ~37B, planned as 50B. Heartbeats are 70% (35B). At ~300 bytes that's 15TB a day raw, 3TB compressed.

Two serving stores. Regional collectors write to Kafka (events.playback keyed by user_id, 7-day retention). Flink feeds ClickHouse for trending and live counts; Spark feeds BigQuery for history and A/B tests; raw events land in S3.

Acknowledge after Kafka. Collectors validate and batch, but answer the client only after Kafka acks: acking earlier would lose buffered events if a collector crashed. 20 collectors per region handle ~10K events a second each.

Trending from unique viewers. Raw counts favour long content and bots, so score unique viewers (HyperLogLog) in a 1-hour window sliding every minute, normalized by a category baseline and boosted by velocity. Scores refresh every 30-60 seconds.

Three freshness tiers. Events under 5 minutes late update real-time numbers; under 24 hours they queue for the T+1 preliminary job at 6 AM; later ones wait for the T+7 final job. Phones persist events in SQLite.

Creator analytics. A star schema (fact_daily_views with five dimensions) partitioned by viewing date and clustered by content. 90% of queries hit the agg_daily_content rollup; HLL sketches merge uniques across days instead of summing them.

• Acknowledging clients before Kafka acks, so a collector crash loses events.
• Counting raw events for trending, which favours long titles and bots.
• Summing daily unique viewers over 30 days, so a 5-day viewer counts 5 times.

Day 1Tenant Isolation Strategies

The isolation spectrum. Shared tables with a tenant_id column, then a schema per tenant, a database per tenant, dedicated infrastructure and separate cloud accounts. Each step adds isolation and cost; no tenant identifier at all is never acceptable.

Tenant from the token. The verified JWT claim is authoritative; a subdomain or X-Tenant-ID header is only a hint and a mismatch is rejected. The tenant lives in a ContextVar, so concurrent requests can't see each other's tenant.

Fail loudly without a tenant. get_current_tenant() raises TenantContextError when no context is set. Any fallback, a default tenant or the previous request's, is exactly how a query ends up without its tenant filter.

RLS as defense in depth. Application repositories add tenant_id to every query; PostgreSQL RLS filters again in the database. FORCE makes policies apply to the table owner too, and app.current_tenant is cleared before a pooled connection is released.

Context must travel. Events carry tenant_id (also the partition key) and jobs take it as an explicit argument, because a ContextVar doesn't reach another process. Cache keys, file paths, search indexes, logs and metrics all include the tenant.

• A query without WHERE tenant_id = ?, returning every tenant's rows.
• Trusting a client-supplied X-Tenant-ID, so an attacker picks the victim's tenant.
• Cache keys like user:{user_id} without a tenant prefix: IDs repeat across tenants.

Day 2Noisy Neighbor Prevention

One tenant breaks everyone. One "export all data" report scanned 2.3 billion rows for 847 seconds: API latency went from 200ms to 15,000ms and 500 other tenants couldn't use the product. Legitimate use, no limits.

Rate limits aren't enough. Rate limiting stops request volume, not expensive operations: five requests a minute can still scan billions of rows. Add concurrency, compute and volume quotas, and enforce at the gateway, application, database and infrastructure.

Limits shared across servers. A token bucket refills up to burst_size: short bursts pass, the long-run rate holds. Per-server limiters give a tenant N times its limit, so one atomic Lua script updates a Redis sliding-window log.

Guard the database. Concurrency slots expire after their TTL, so a crashed worker can't leak one forever. EXPLAIN-based analysis rejects queries over 10 million estimated rows unless the plan is Enterprise; per-tenant pools set statement timeouts.

Schedule fairly. Round-robin across per-tenant queues: a tenant with 100 queued jobs can't make another wait for all of them, and each queue is capped. Weighted fair queuing (free 1, pro 5, enterprise 20) gives paying tiers a larger share.

• One global limiter: a tenant at 900 req/sec leaves 100 for everyone else.
• Throttling silently by sleeping, which hides the limit and holds connections; return 429 with Retry-After.
• One limit for every operation: 1,000 exports melt the database.

Day 3Data Residency and GDPR

Four words to keep apart. Residency is where data is stored; sovereignty is which country's laws apply; localization is a legal requirement to keep data in-country; a transfer is moving data across borders, which needs a legal basis.

What counts as personal data. Names, emails, IP addresses, cookie and device IDs, location and behaviour. Pseudonymized data is still personal data, and backups and caches follow the same rules as the primary data.

Regional data, global control plane. The common SaaS pattern: each tenant is assigned a region at onboarding, and its database, storage and search live there. The global control plane holds only metadata: tenant registry, configuration, routing rules.

Consent done right. Freely given, specific per purpose, informed, unambiguous, and as easy to withdraw as to give. Keep an append-only log (exact text, version, IP, time) plus a current-status table; never asked counts as not granted.

Transfers need a basis. Adequacy decision first, then Standard Contractual Clauses with supplementary measures, or BCRs for corporate groups. Post-Schrems II, document a Transfer Impact Assessment and review it, because adequacy can be revoked.

• Keeping EU data in Frankfurt but sending backups to a US bucket.
• Logging emails or user IDs to a global logging service.
• Using US vendors without a DPA and a transfer mechanism, or assuming adequacy is permanent.

Day 4Right to Deletion

What Article 17 asks. Respond within one month (two more for complex cases), delete every copy including backups eventually, and have processors delete too. You may refuse for legal obligations, public interest or legal claims, and must explain why.

Deleting is not marking. Soft delete leaves the data in storage and backups, so it's only a step before hard delete. Pseudonymized data is still personal; truly anonymous aggregates ("users made 10,000 orders") may be kept.

Inventory, then orchestrate. Map every place personal data lives, then run one executor per system in dependency order: search, files, analytics, third parties, the primary database (it holds the user_id mapping and foreign keys), and caches last.

Stop, retry, verify. Targets at the same priority run in parallel; after a failed group the run stops and a retry resumes at that point. The request completes only after every system is re-checked and the data is confirmed gone.

Keep what the law requires. Orders are financial records, so they're anonymized (user_id NULL, addresses 'DELETED'), not deleted. Shared content shows "Deleted User". The deletion audit trail is kept, with the user ID redacted, as proof.

Backups expire, restores re-delete. You can't remove one user from a backup, so backups expire under a retention policy that's tracked. If a backup is restored first, the pending deletions are re-applied before the data goes live.

• Deleting the users row and forgetting search, analytics, caches and third parties.
• Reporting "deleted" without a verification pass over every system.
• Deleting audit logs with the user, leaving no proof of what happened.

Day 5Security Architecture

Defense in depth. Five layers, each assuming the others might fail: perimeter (WAF, DDoS protection, rate limits), network (VPC, private subnets), application (authn, authz, validation), data (encryption, keys), monitoring (audit logs, anomaly detection).

Zero trust. Never trust, always verify: network location grants nothing, every connection is authenticated and every request authorized. Least privilege, assume breach, micro-segmentation. Google built BeyondCorp after Operation Aurora.

Secrets never in code. Not in code, config files, container images or environment variables. Fetch from Vault or Secrets Manager at connection time so rotation needs no restart. A leaked secret is already copied everywhere: rotate it.

Encrypt at every layer. TLS 1.2+ externally and mTLS between services; TDE and SSE-KMS at rest; field-level encryption for SSNs and cards, because TDE decrypts for any query. A key hierarchy (HSM master, tenant keys, data keys) limits blast radius.

Revocable tokens. 15-minute access JWTs and 7-day refresh tokens, plus a server-side session checked on every request, so deleting the session revokes access at once. Five failed logins mean a 15-minute lockout; errors stay generic.

• Trusting frontend validation: amount=-500 sent straight to the API adds money.
• Fetching documents by ID alone, without AND tenant_id = ? from the authenticated user.
• allow_origins=["*"] with credentials, or logging passwords, card numbers and auth headers.

BossDesign a Multi-Tenant Enterprise SaaS Platform

Size it first. ~100M documents at 500KB is 50TB; with metadata 60TB, 180TB with 3x replication. Extracted text of 10KB each is a 1TB index, ~3TB in Elasticsearch. 1,000 requests a second at peak, 60% of them searches.

Regional data planes, global control plane. US, EU and APAC each run a full stack: API, PostgreSQL, Elasticsearch, S3, Redis, Kafka. The global control plane holds only the tenant registry, flags, quotas and routing. No tenant data crosses regions.

Five isolation layers. Tenant from the verified JWT (the subdomain must match); repositories that add tenant_id; RLS in PostgreSQL; a search index per enterprise tenant or a filtered shared index; S3 paths and IAM by tenant, with per-tenant keys.

Quotas and fair queues. Per-plan quotas on API calls, searches, uploads and query timeouts. Each tenant gets its own capped job queue; dividing the plan weight by (1 + depth * 0.1) shrinks a deep queue's share.

Uploads go straight to S3. The upload service issues a presigned URL for the tenant's regional bucket, so bytes skip the API servers; text extraction and thumbnails run asynchronously from a queue. The server picks the bucket, never the client.

• Reading tenant_id from the request body so admins can "switch tenants".
• Putting user emails in the global control plane for faster login routing.
• Deleting the primary database first, leaving search, file and analytics copies you can no longer find.

Day 1SLIs, SLOs and SLAs

Measure, target, promise. An SLI is the measurement (share of requests under 200ms), an SLO the internal target (99.9% over 30 days), an SLA the promise with consequences (99.5%, or a 10% credit). SLO stricter than SLA.

Specify the SLI precisely. Good events over valid events, measured at the load balancer for the client's view. Exclude health checks and planned maintenance; count 4xx as good, because the service handled them correctly.

Appropriate reliability. 100% is impossible, infinitely expensive and leaves no room to ship. Going from 99.9% to 99.99% may cost ~10x infrastructure and slow features, yet most users can't tell. Don't set the SLO at current performance: leave headroom.

Error budgets drive decisions. 99.9% over 30 days allows 43.2 minutes of failure. Above 50% left, ship freely; 20-50%, no risky deploys; under 20%, pause features; exhausted, freeze. The policy must be written down and enforced.

Alert on burn rate. Burn rate is the actual error rate over the allowed one; 1x empties the budget exactly at the window's end. Page only when a short and a long window both exceed it, so blips don't page.

• SLOs on server metrics like CPU < 70%: requests can fail at 50% CPU.
• Setting the SLO at current performance, so normal variation keeps breaching it.
• Fifty SLOs instead of 3-5 user-facing ones, or SLOs with no error budget policy.

Day 2Observability

Monitoring vs observability. Monitoring watches for known failure modes with pre-defined dashboards and alerts: it tells you when. Observability lets you ask questions you didn't anticipate, like slicing slow requests by app version: it tells you why. You need both.

Three pillars, one trace ID. Metrics say when and how bad, logs what happened, traces where in the request path. A trace ID propagated through every service and written in every log, plus exemplars on metrics, turns hours into minutes.

Pick the right metric type. Counters only go up (use rate), gauges go up and down, histograms count observations in buckets. Histograms add up across pods for a fleet-wide p99; summaries pre-compute per-pod quantiles that can't be aggregated.

Keep labels bounded. Total series is the product of label cardinalities: method × endpoint × status is 10,000 series, but adding user_id for 1M users makes 10 billion. Per-user detail belongs in logs and traces.

RED for services, USE for resources. Rate, errors and duration for APIs; utilization, saturation and errors for databases, queues and disks. Dashboards show the four golden signals first and compare with yesterday and last week.

Propagate the context. Service B continues A's trace only if A injects the context (traceparent or B3 headers) and B extracts it. Otherwise each service starts its own trace and the journey is lost.

• Alerting on causes like CPU > 80% instead of symptoms users feel, such as p99 latency.
• Logs without context ("Payment failed") or with secrets: no passwords, full cards or tokens.
• Alerts without runbooks, or error_count > 0, which is pure noise.

Day 3Deployment Strategies

Small and frequent. The safest deployment is a small one; the safest rollback is a fast one. Deploy daily in small increments, early in the week, watch dashboards during the deploy, and declare success only after the bake time.

Rolling and blue-green. Rolling replaces a few instances at a time: no downtime, but slow rollback, and both versions share the database and APIs. Blue-green switches all traffic at once; rollback is instant because the old side still runs.

Canary with automated analysis. Shift 1%, 10%, 50%, then 100%, watching each stage for 15 minutes. Roll back if errors rise over 1% absolute or p99 over 10% versus stable; under 100 requests, fail closed, never promote.

Decouple deploy from release. Ship code with the flag off, then enable it gradually. Hashing flag_key:user_id into a bucket 0-99 keeps each user on one path. Remove release flags after rollout; keep kill switches.

Expand, migrate, contract. To rename a column: add the new one and write both; backfill, then switch reads; drop the old column only when no running code uses it. Old and new code run together during every deploy.

• Renaming a column in place while old and new code both run.
• Schema and code changes in one deploy, so neither can roll back alone.
• No rollback plan, or a deploy on Friday followed by walking away.

Day 4Capacity Planning

The capacity cycle. Measure what you can handle now, forecast demand (growth, seasonality, events), find what breaks first, plan the scaling, validate with load tests, then measure again. Scale before you need to: projects take 4-6 weeks.

Headroom. Headroom = (max capacity − current) / max capacity. A database good for 3,000 QPS at 2,000 has 33%. Keep at least 30% on critical resources: traffic is bursty, retries storm, GC pauses happen, and 100% means already failing.

Capacity is the sustainable max. Stress-test the latency curve: in the example, latency stays within the SLO up to 1,000 RPS and errors start at 1,800. Capacity is 1,000, not the breaking point; run at most ~700.

Little's Law. L = λ × W. 500 RPS at 200ms needs 100 concurrent workers: 25 pods at 4 workers each, ~36 with 30% headroom. Past capacity, requests queue, W grows, L grows, and latency spirals.

Stay below 70-80%. In a single-server queue, wait ≈ service time × ρ / (1 − ρ): about 1x at 50% utilization, 4x at 80%, 9x at 90%, infinite at 100%. Never run critical resources above 70-80%.

Pre-scale known events. Auto-scaling takes 3-5 minutes, so a sudden spike lands first. Pre-scale for launches and campaigns, scale up fast and down slowly (+3 pods vs −1 with a 15-minute cooldown), and plan for peaks, not averages.

• Waiting for problems: by the time you see them, users are affected.
• Trusting averages: 30% average CPU can hide an evening peak near 95%.
• Load testing only in staging, or ignoring the database and third-party limits behind you.

Day 5Incident Management

Severity sets the response. SEV1, complete outage over $10,000 an hour: all hands, execs, status page. SEV2, a major degradation: on-call plus backup. SEV3 waits for business hours; SEV4 goes to the ticket queue. Not every alert is an incident.

Mitigate first. Detect, triage, respond, mitigate, resolve, learn. Stop user impact before chasing the root cause: recent deploy, roll back; dependency down, fail over or degrade; overloaded, scale up or rate limit.

Clear roles. The Incident Commander coordinates and decides but doesn't debug; the Technical Lead diagnoses and fixes; the Communications Lead runs updates and the timeline. One change at a time, announced first.

Communicate on a clock. First status page update within 10 minutes; SEV1 updates at least every 15 minutes, even "still investigating". Plain, impact-focused language; never blame a provider you chose.

Blameless postmortems. Within 48 hours. Ask why the system allowed it, never stop at human error, and use the five whys until you reach something fixable. Action items get owners and dates and are tracked like any work.

Sustainable on-call. At least 8 people in rotation, fewer than 5 pages per week-long shift, handoff meetings between shifts, and only actionable alerts with runbooks. 50 noisy pages a week train people to ignore the real one.

• Too many cooks: several people changing things at once.
• Going silent while heads down, so customers learn from Twitter.
• No postmortem, or one that stops at "the engineer forgot to test".

BossThe Grand Finale

Size the bidding platform. 10 billion requests a day is ~116,000 a second, ~500,000 at peak. At ~8 KB of audit data each: 80 TB a day, 29 PB a year, ~200 PB over 7 years, so tiered storage.

A 100ms latency budget. Receive 2ms, parse 5ms, DSP fan-out 75ms, auction 10ms, respond 3ms: 95ms, with 5ms to spare. The 70ms DSP timeout is a hard deadline inside the fan-out; a late answer loses the impression.

Budget leases, never overspend. A global coordinator splits each budget across regions, keeping a reserve; regions lease small amounts to pods with an atomic Redis check-and-decrement. Pods spend locally, so no network call per bid; no lease means no bid.

Fail toward under-spending. If a region fails, its leases expire after 30 seconds and only provably unspent budget is released; the rest stays locked until reconciliation. In a partition a region bids on existing leases, then stops.

Fan out with breakers. Query 5-10 DSPs in parallel under one global deadline and auction whatever arrived. A per-DSP breaker opens after 5 consecutive failures and retries after 30 seconds. Under overload, query the top DSPs and shed low-value requests.

Audit through an outbox. Write the bid event to an outbox in the budget update's transaction; Debezium ships it to Kafka keyed by advertiser_id for ordered billing. Hot 7 days, warm Parquet to 90 days, Glacier to 7 years.

• Each server checking "is there budget?" alone: $10 left becomes $19,990 overspent.
• Publishing audit events straight to Kafka beside the budget write: a dual write can lose them.
• Waiting on slow DSPs instead of one hard deadline and circuit breakers.