Day 1SLIs, SLOs and SLAs

Measure, target, promise. An SLI is the measurement (share of requests under 200ms), an SLO the internal target (99.9% over 30 days), an SLA the promise with consequences (99.5%, or a 10% credit). SLO stricter than SLA.

Specify the SLI precisely. Good events over valid events, measured at the load balancer for the client's view. Exclude health checks and planned maintenance; count 4xx as good, because the service handled them correctly.

Appropriate reliability. 100% is impossible, infinitely expensive and leaves no room to ship. Going from 99.9% to 99.99% may cost ~10x infrastructure and slow features, yet most users can't tell. Don't set the SLO at current performance: leave headroom.

Error budgets drive decisions. 99.9% over 30 days allows 43.2 minutes of failure. Above 50% left, ship freely; 20-50%, no risky deploys; under 20%, pause features; exhausted, freeze. The policy must be written down and enforced.

Alert on burn rate. Burn rate is the actual error rate over the allowed one; 1x empties the budget exactly at the window's end. Page only when a short and a long window both exceed it, so blips don't page.

• SLOs on server metrics like CPU < 70%: requests can fail at 50% CPU.
• Setting the SLO at current performance, so normal variation keeps breaching it.
• Fifty SLOs instead of 3-5 user-facing ones, or SLOs with no error budget policy.

Day 2Observability

Monitoring vs observability. Monitoring watches for known failure modes with pre-defined dashboards and alerts: it tells you when. Observability lets you ask questions you didn't anticipate, like slicing slow requests by app version: it tells you why. You need both.

Three pillars, one trace ID. Metrics say when and how bad, logs what happened, traces where in the request path. A trace ID propagated through every service and written in every log, plus exemplars on metrics, turns hours into minutes.

Pick the right metric type. Counters only go up (use rate), gauges go up and down, histograms count observations in buckets. Histograms add up across pods for a fleet-wide p99; summaries pre-compute per-pod quantiles that can't be aggregated.

Keep labels bounded. Total series is the product of label cardinalities: method × endpoint × status is 10,000 series, but adding user_id for 1M users makes 10 billion. Per-user detail belongs in logs and traces.

RED for services, USE for resources. Rate, errors and duration for APIs; utilization, saturation and errors for databases, queues and disks. Dashboards show the four golden signals first and compare with yesterday and last week.

Propagate the context. Service B continues A's trace only if A injects the context (traceparent or B3 headers) and B extracts it. Otherwise each service starts its own trace and the journey is lost.

• Alerting on causes like CPU > 80% instead of symptoms users feel, such as p99 latency.
• Logs without context ("Payment failed") or with secrets: no passwords, full cards or tokens.
• Alerts without runbooks, or error_count > 0, which is pure noise.

Day 3Deployment Strategies

Small and frequent. The safest deployment is a small one; the safest rollback is a fast one. Deploy daily in small increments, early in the week, watch dashboards during the deploy, and declare success only after the bake time.

Rolling and blue-green. Rolling replaces a few instances at a time: no downtime, but slow rollback, and both versions share the database and APIs. Blue-green switches all traffic at once; rollback is instant because the old side still runs.

Canary with automated analysis. Shift 1%, 10%, 50%, then 100%, watching each stage for 15 minutes. Roll back if errors rise over 1% absolute or p99 over 10% versus stable; under 100 requests, fail closed, never promote.

Decouple deploy from release. Ship code with the flag off, then enable it gradually. Hashing flag_key:user_id into a bucket 0-99 keeps each user on one path. Remove release flags after rollout; keep kill switches.

Expand, migrate, contract. To rename a column: add the new one and write both; backfill, then switch reads; drop the old column only when no running code uses it. Old and new code run together during every deploy.

• Renaming a column in place while old and new code both run.
• Schema and code changes in one deploy, so neither can roll back alone.
• No rollback plan, or a deploy on Friday followed by walking away.

Day 4Capacity Planning

The capacity cycle. Measure what you can handle now, forecast demand (growth, seasonality, events), find what breaks first, plan the scaling, validate with load tests, then measure again. Scale before you need to: projects take 4-6 weeks.

Headroom. Headroom = (max capacity − current) / max capacity. A database good for 3,000 QPS at 2,000 has 33%. Keep at least 30% on critical resources: traffic is bursty, retries storm, GC pauses happen, and 100% means already failing.

Capacity is the sustainable max. Stress-test the latency curve: in the example, latency stays within the SLO up to 1,000 RPS and errors start at 1,800. Capacity is 1,000, not the breaking point; run at most ~700.

Little's Law. L = λ × W. 500 RPS at 200ms needs 100 concurrent workers: 25 pods at 4 workers each, ~36 with 30% headroom. Past capacity, requests queue, W grows, L grows, and latency spirals.

Stay below 70-80%. In a single-server queue, wait ≈ service time × ρ / (1 − ρ): about 1x at 50% utilization, 4x at 80%, 9x at 90%, infinite at 100%. Never run critical resources above 70-80%.

Pre-scale known events. Auto-scaling takes 3-5 minutes, so a sudden spike lands first. Pre-scale for launches and campaigns, scale up fast and down slowly (+3 pods vs −1 with a 15-minute cooldown), and plan for peaks, not averages.

• Waiting for problems: by the time you see them, users are affected.
• Trusting averages: 30% average CPU can hide an evening peak near 95%.
• Load testing only in staging, or ignoring the database and third-party limits behind you.

Day 5Incident Management

Severity sets the response. SEV1, complete outage over $10,000 an hour: all hands, execs, status page. SEV2, a major degradation: on-call plus backup. SEV3 waits for business hours; SEV4 goes to the ticket queue. Not every alert is an incident.

Mitigate first. Detect, triage, respond, mitigate, resolve, learn. Stop user impact before chasing the root cause: recent deploy, roll back; dependency down, fail over or degrade; overloaded, scale up or rate limit.

Clear roles. The Incident Commander coordinates and decides but doesn't debug; the Technical Lead diagnoses and fixes; the Communications Lead runs updates and the timeline. One change at a time, announced first.

Communicate on a clock. First status page update within 10 minutes; SEV1 updates at least every 15 minutes, even "still investigating". Plain, impact-focused language; never blame a provider you chose.

Blameless postmortems. Within 48 hours. Ask why the system allowed it, never stop at human error, and use the five whys until you reach something fixable. Action items get owners and dates and are tracked like any work.

Sustainable on-call. At least 8 people in rotation, fewer than 5 pages per week-long shift, handoff meetings between shifts, and only actionable alerts with runbooks. 50 noisy pages a week train people to ignore the real one.

• Too many cooks: several people changing things at once.
• Going silent while heads down, so customers learn from Twitter.
• No postmortem, or one that stops at "the engineer forgot to test".

BossThe Grand Finale

Size the bidding platform. 10 billion requests a day is ~116,000 a second, ~500,000 at peak. At ~8 KB of audit data each: 80 TB a day, 29 PB a year, ~200 PB over 7 years, so tiered storage.

A 100ms latency budget. Receive 2ms, parse 5ms, DSP fan-out 75ms, auction 10ms, respond 3ms: 95ms, with 5ms to spare. The 70ms DSP timeout is a hard deadline inside the fan-out; a late answer loses the impression.

Budget leases, never overspend. A global coordinator splits each budget across regions, keeping a reserve; regions lease small amounts to pods with an atomic Redis check-and-decrement. Pods spend locally, so no network call per bid; no lease means no bid.

Fail toward under-spending. If a region fails, its leases expire after 30 seconds and only provably unspent budget is released; the rest stays locked until reconciliation. In a partition a region bids on existing leases, then stops.

Fan out with breakers. Query 5-10 DSPs in parallel under one global deadline and auction whatever arrived. A per-DSP breaker opens after 5 consecutive failures and retries after 30 seconds. Under overload, query the top DSPs and shed low-value requests.

Audit through an outbox. Write the bid event to an outbox in the budget update's transaction; Debezium ships it to Kafka keyed by advertiser_id for ordered billing. Hot 7 days, warm Parquet to 90 days, Glacier to 7 years.

• Each server checking "is there budget?" alone: $10 left becomes $19,990 overspent.
• Publishing audit events straight to Kafka beside the budget write: a dual write can lose them.
• Waiting on slow DSPs instead of one hard deadline and circuit breakers.