Day 1Timeout Hell

Slow is worse than down. A down service fails fast; a slow one holds your threads. With 100 threads, a 10 s downstream cuts you from 1,000 to 10 requests a second. Timeouts turn slow into down.

Percentiles, not averages. Set a timeout at the ~65 ms average and 5% of healthy requests fail. Rule of thumb: timeout = P99 × 2–3, so P99 = 200 ms gives 400–600 ms.

Timeout budgets. Give each dependency a share of the SLA, not the whole SLA. For a 5 s payment: 4.5 s budget, fraud check 500–600 ms, bank 3.5 s, notification optional with ~300 ms.

Deadline propagation. Pass the absolute deadline down the chain, so each hop waits only for the time left. Per-hop 5 s timeouts across three services can add up to 15 s after the user gave up at 5.

Connection vs read timeout. Connection timeout covers the TCP handshake and should be short (1–5 s): if you can't connect, something is badly wrong. Read timeout covers the work and depends on the operation.

• No timeout, or 30 s "to be safe": threads pile up and you look dead.
• Timeout = average or P99, failing healthy requests every day.
• Adaptive timeouts without alerting on the timeout value and a hard maximum.

Day 2Idempotency in Practice

Request lost or response lost? A client can't tell them apart: both look like a timeout. No timeout tuning fixes that; the only answer is making the operation safe to retry.

The idempotency key. The client sends a unique key per logical operation. The server records it with status "processing" before calling the bank, stores the result, and returns the cached result for any repeat.

Pass the key downstream. Send your idempotency key to the bank too: it deduplicates as well, and on an ambiguous timeout you can ask it what happened to that key, then update your record.

Guard the edge cases. Hash the request body so a reused key with a different amount is rejected. Only completed requests block retries; a failed one may run again. Extend the TTL on completion.

• POST payments with no idempotency key: a double-click charges twice.
• A TTL shorter than the retry window, so a late retry processes again.
• Marking a partial failure "failed": a retry charges the customer twice.

Day 3Circuit Breakers

Fail fast when it's clearly broken. Timeouts and retries still wait 3.5 s per call and pile load onto a failing bank. A circuit breaker stops calling: users get an error in ~10 ms and the bank can recover.

Closed, open, half-open. Closed: calls pass and failures are counted. Open: calls fail immediately until a timer expires (often 30–60 s). Half-open: one test call decides, closed on success or open again.

Count what's really a failure. Timeouts, 5xx and refused connections count. 4xx errors and business rejections like "card declined" don't: the service worked correctly and said no.

Use failure rate, not count. "5 failures in 60 s" trips on Black Friday at 10,000 req/s even though 0.05% is normal. Open at a failure rate (around 50%) with a minimum number of calls.

Retry, breaker and bulkhead together. Check the circuit breaker first (cheapest), take a bulkhead slot second, retry transient failures last, with exponential backoff and jitter. They're layers, not alternatives.

• Closing to full traffic at once: the backed-up herd crashes the recovering service. Ramp up.
• Alerting only on error rate: an open circuit lowers it and hides the outage.
• A fallback for payments, auth or inventory: you can't fake those.

Day 4Webhook Delivery System

At-least-once plus idempotent receivers. True exactly-once is impossible (the Two Generals problem). Retry until acknowledged and give every event an ID the receiver uses as its idempotency key; a duplicate costs one lookup.

The pipeline. Event sources publish to a durable bus (Kafka); a dispatcher fans each event out to its subscriptions; delivery queues partitioned by subscription feed workers; exhausted retries go to a dead letter queue.

Retry what can succeed. Retry network errors, timeouts, 5xx and 429 (honour Retry-After). A 4xx means the request is malformed: retrying won't help, so log it or dead-letter it.

Sign and timestamp every webhook. HMAC-SHA256 over the exact bytes sent, with a timestamp inside the signature. Receivers compare in constant time and reject anything more than 5 minutes old, which stops replays.

Recover fast after an outage. Monitor endpoint health. When a dead endpoint recovers, move its pending retries to an immediate queue and ramp up gradually, within its rate limit, instead of waiting out the backoff.

• Re-serializing the payload after signing, so the receiver's HMAC over the raw body fails.
• Retrying 4xx responses that will never succeed.
• Delivering events for one entity in parallel: partition by entity to keep order.

Day 5Distributed Cron & Job Scheduling

Locks alone don't work. A plain lock stays held when its server crashes. A lock timeout lets a slow job run twice. A Redis lock released with GET then DELETE can delete the next holder's lock.

Perfect is impossible, so be practical. FLP says consensus can't be guaranteed with even one faulty process. Production cron runs at-least-once, makes jobs idempotent, tolerates small delays and alerts on missed runs.

One leader schedules. Elect a leader with a lease in etcd, ZooKeeper or Consul; followers wait to take over. Use a coordination service instead of building your own unless you build databases.

Fencing tokens stop stale leaders. Each new leader gets a higher token and sends it with every operation; resources reject older tokens. A deposed leader paused by GC can't do damage when it wakes up.

Track jobs outside the leader. Record executions in a job store with heartbeats. A new leader skips jobs still beating, a stuck-job detector reschedules silent ones, and checkpoints let a job resume instead of redoing work.

• A lock with no expiry: one crash and the job never runs again.
• Releasing a lock you may no longer own; check and delete atomically.
• Non-idempotent jobs: a failover rerun sends the same 50 emails again.

BossThe Ultimate System Design Interview

The spike is the problem. 5 million users hit one 10-minute sale and 80% of traffic lands in the first 30 seconds: ~150,000 requests a second at peak. CDN, WAF and rate limits absorb the herd before it reaches services.

Atomic claims in Redis. A Lua script checks the user's existing claim and decrements the inventory counter in one atomic step, restoring it if it goes below zero. No database in the hot path, so no overselling.

Claims expire back into the pool. A claim holds a unit for 3 minutes. A claim-expiry worker returns unpaid units to inventory, triggering a second wave of claims (~15,000 claim operations for 10,000 units).

Checkout uses every Week 2 pattern. Validate the claim, check the idempotency store by claim_id, call Stripe behind a circuit breaker with a timeout and its idempotency key, create the order in a transaction, publish events asynchronously.

Precompute notifications. Build the 2.4M-recipient list 10 minutes before the sale into Redis batches. At start, 24 workers fan out at once and queue everything in ~10 seconds; push and email have separate pools.

• Checking stock then decrementing in two steps: a race that oversells.
• Computing millions of notification recipients at the moment the sale starts.
• Putting the database in the claim path at 40,000 claim attempts a second.