Every service you've ever designed ends with "and we'd monitor it." This question asks you to build the monitor: the thing that collects millions of numbers a second from every host, draws the dashboards, and wakes someone at 3 a.m. when error rates climb. It has to keep working on the worst day your other systems have, because that's the day you need it most.
The metrics-platform interview end to end: 100,000 hosts and 50 million series give 5 million points a second; compression takes a point from 16 bytes to about 1.5, so a raw week is 4.5 TB, not 48. Agents push through a gateway into Kafka, writers keep a 2-hour head and flush blocks to S3, and queries use a label index and rollups. Deep dives cover delta-of-delta and XOR compression, the cardinality trap, and an alert path of its own: pending then firing, grouping 200 alerts into one page, and a watchdog for the watcher.
Phase 1: Requirements (5 minutes)#
"Metrics only, or logs and traces too?"
"Metrics. Numbers over time with labels, like http_requests_total{service="checkout", status="500"}."
"And how long do we keep them, at what resolution?"
"Engineers want 10-second detail for the last week, and trends for a year."
- Ingest counters, gauges and histograms from agents and services, with labels
- Query by metric and label filters, with functions (
rate,sum by, percentiles) over a time range - Dashboards that refresh every few seconds
- Alert rules (
error rate > 5% for 5 minutes) evaluated continuously - Notifications to on-call (pager, chat, email) with grouping, silencing and escalation
- Retention: 10 s raw for 7 days, 1-minute rollups for 30 days, 1-hour rollups for a year
- Write-heavy: millions of points per second, far more writes than reads
- Fresh: a point is queryable within ~10 seconds of being emitted
- Alerting is the priority: dashboards may degrade, alerting must not
- Cheap storage: a year of history without a petabyte bill
- Reads: recent data in under a second; a year-long query in a few seconds
Phase 2: Estimation (5 minutes)#
- Hosts: 100,000, each with ~500 time series (system metrics + its services' metrics)
- Active series: 50M
- Collection interval 10 s → 50M ÷ 10 = 5M points/s
- A raw point is a timestamp + a float = 16 bytes; with time-series compression (delta-of-delta timestamps, XOR'd floats) about 1.5 bytes. Facebook's Gorilla paper (2015) reported about 1.37 bytes per point.
- Raw per day: 5M × 86,400 ≈ 432 billion points × 1.5 B ≈ 650 GB/day → 7 days ≈ 4.5 TB
- 1-minute rollups (min, max, sum, count): 50M × 1,440 × ~6 B ≈ 430 GB/day → 30 days ≈ 13 TB
- 1-hour rollups: 50M × 24 × ~6 B ≈ 7 GB/day → 1 year ≈ 2.6 TB
- Without compression the raw week alone would be about 48 TB
"Two numbers drive the design. 5M points a second means a write path built for appends, not a general database. And compression is worth 10x in storage, so the storage format is a deep dive."
Phase 3: API (3 minutes)#
POST /v1/write (agents, batched every 10 s, compressed)
[ { "series": "http_requests_total{service=\"checkout\",status=\"500\"}", "points": [[1761400000, 1532]] }, ... ]
→ 204
GET /v1/query_range?q=sum by (status)(rate(http_requests_total{service="checkout"}[5m]))&start=...&end=...&step=60s
→ 200 { "series": [ { "labels": {...}, "points": [[t, v], ...] } ] }
PUT /v1/alerts/checkout-errors
{ "expr": "rate(errors[5m]) / rate(requests[5m]) > 0.05", "for": "5m", "severity": "page", "route": "team-checkout" }Phase 4: High-level design (10 minutes)#
%% title: Metrics platform: buffered ingest, a time-series store, and an alerting path of its own
flowchart TD
AG@{ icon: "sd:servers", label: "Agents on 100K hosts<br/>batch every 10 s" }
GW@{ icon: "sd:gateway", label: "Ingest gateway<br/>auth · limits · validate" }
K@{ icon: "tech:kafka", label: "Kafka<br/>partitioned by series ID" }
TS["TSDB writers<br/>in-memory head · compressed blocks"]:::core
OS@{ icon: "tech:s3", label: "Object storage<br/>blocks + rollups" }
Q["Query service<br/>index · cache · rollups"]:::service
GF@{ icon: "tech:grafana", label: "Dashboards" }
AE["Alert evaluator<br/>rules every 30 s"]:::service
AM@{ icon: "sd:alert", label: "Alert manager<br/>group · dedupe · route" }
PG@{ icon: "sd:bell", label: "Pager / chat / email" }
AG --> GW
GW --> K
K --> TS
TS --> OS
Q --> TS
Q --> OS
GF --> Q
K -->|"own consumer"| AE
AE --> AM
AM --> PG
class AG client
class GW edge
class K queue
class OS store
class AM,PG external- Collect. The agent reads counters every 10 s, batches them, compresses the batch and sends it to the gateway.
- Buffer. The gateway checks auth and per-tenant limits, then appends to Kafka, partitioned by series ID so each series lands on one writer. If the store is slow, Kafka absorbs the backlog.
- Store. Writers keep the last ~2 hours of each series in memory (the "head"), compress it into blocks, and flush blocks to disk and object storage. A write-ahead log covers crashes.
- Query. The query service finds matching series through a label index, reads the head for recent data and blocks or rollups for older data, and caches results.
- Alert. The evaluator consumes the same stream (or queries the head) every 30 s, and hands firing alerts to the alert manager, which groups, dedupes and routes them.
Pull (scrape)
- The server fetches
/metricsfrom each target every 10 s (the Prometheus model) - A target that stops answering is itself a signal ("up = 0")
- Needs service discovery, and reaching every target
- Awkward for short-lived jobs and hosts behind NAT
Push (agents)
- Agents send batches to a gateway (the Datadog-style model)
- Works through firewalls, for short jobs and at very large scale
- Needs a heartbeat to tell "quiet" from "dead"
- The gateway must rate-limit misbehaving senders
"At 100,000 hosts I'd push through a gateway into Kafka: it decouples senders from storage and gives me a buffer. Each agent also sends a heartbeat series, so a silent host still raises an alert."
Phase 5: Deep dives (15 minutes)#
Deep dive 1: Storing 5 million points a second#
A general-purpose database would store each point as a row: 16 bytes of data plus index and row overhead, with a B-tree insert for every point. Time-series stores win by exploiting three facts: points arrive in time order, they are never updated, and neighbouring values are similar.
Series, not rows
A series is a metric name plus its labels. Each gets an ID, and its points are stored together in time order: a 2-hour chunk per series.
Compression
Timestamps arrive every ~10 s, so store the difference of differences: usually 0, which takes 1 bit. Values change little, so XOR each float with the previous one and store only the changed bits. Together that's 16 bytes down to about 1.5.
An inverted index on labels
service="checkout" → the set of series IDs with that label. A query intersects these sets (like a search engine's postings lists), then reads only those series' chunks.
Rollups
A background job downsamples old blocks into 1-minute and 1-hour rollups that keep min, max, sum and count, so averages stay correct and spikes stay visible. A year-long dashboard reads 8,760 hourly points per series, not 3 million raw ones.
The cardinality trap
A developer adds a user_id label to http_requests_total on a service with 10 million users. What happens to the store?
One strong answer
- Series count explodes. Every distinct label combination is a new series: up to 10 million new series for one metric, more than the 50M budget allows for everything else combined, each with its own index entries and in-memory head.
- Memory goes first. Writers hold a head chunk per active series, so memory, not disk, runs out. Queries that touch the metric slow down because their index sets are huge.
- Defence: per-tenant and per-metric series limits at the gateway (reject or drop the label beyond a cap), dashboards that show the top series-creating metrics, and the rule from the course: labels must have bounded values. Per-user detail belongs in logs or traces.
Deep dive 2: Alerting that works on the worst day#
The alerting path is the reason the system exists. It needs its own priorities:
Evaluate close to fresh data
Rules run every 30 s against the in-memory head (or their own Kafka consumer), not through the dashboard query path. A slow dashboard query can never delay a page.
for clauses and states
An alert goes pending when its condition first holds, firing only if it still holds after its for duration (say 5 minutes), and resolved when it clears. One bad scrape doesn't page anyone.
Group, dedupe, silence, route
If 200 hosts lose a disk at once, on-call gets one notification ("200 hosts: disk full"), not 200. Silences cover planned maintenance; routes send checkout alerts to the checkout rota and escalate if nobody acknowledges.
Deliver like a notification system
Each notification is retried with backoff and sent through a second channel if the first fails. Delivery is at least once; the alert key makes duplicates harmless.
Watch the watcher
An always-firing "watchdog" alert is sent to an external dead man's switch every minute. If it stops arriving, the outside service pages: the monitoring system itself is down.
Missing data needs explicit rules too: absent(up{job="payments"}) fires when a service stops reporting at all, which a threshold on its error rate never would.
Phase 6: Trade-offs and failure modes (5 minutes)#
TSDB writers fall behind
Kafka buffers the backlog; dashboards show slightly old data. Alerts read from their own consumer, so they keep up or degrade separately, and lag on that consumer is itself alerted on.
A writer crashes
Its head chunks are rebuilt from the write-ahead log, or by replaying its Kafka partitions from the last flushed offset.
Cardinality explosion
Gateway limits drop new series beyond the cap and alert the owning team, so one bad deploy can't take the platform down for everyone.
Expensive queries
A year-long sum by over millions of series: route it to rollups, cap the series a query may touch, cache repeated dashboard queries for the refresh interval.
Rollups hide spikes
An hourly average flattens a 30-second outage. Keeping max (and percentiles from histograms) in each rollup keeps spikes visible.
| Decision | We chose | We gave up |
|---|---|---|
| Collection | Push via gateway + Kafka buffer | "up = 0" for free from scraping |
| Storage | Compressed per-series chunks + label index | Arbitrary SQL over raw points |
| Retention | Raw 7 d, 1-min 30 d, 1-h 1 y | 10-second detail after a week |
| Alerting | Separate path, for clauses, watchdog |
Simplicity of one shared query path |
Go deeper in the course#
- Week 10, Day 2: Observability: metric types, naming and labels (and why high-cardinality labels hurt), RED and USE, dashboards and alert design
- Week 10, Day 1: SLIs, SLOs and SLAs: error budgets and burn-rate alerts, the rules this system evaluates
- Week 10, Day 5: Incident management: what happens after the page
- Week 8, Day 1: Event ingestion at scale: gateways, batching and Kafka in front of storage
- Week 8, Day 5: Query layer and optimization: pre-aggregation, materialized views and caching, the idea behind rollups
- Week 8, Day 2: Streaming vs batch: evaluating alerts on the stream vs downsampling in batch
- Week 3, Day 3: Backpressure and flow control: what Kafka absorbs when writers slow down
- Week 6, Day 2: Core notification flow: retries, channels and deduplication for delivering alerts
Check yourself#
Metrics monitoring
Q1/4 Why do time-series stores compress points from 16 bytes to about 1.5?
Answer & explanation
Answer: C
Points arrive at regular intervals and change little, so most timestamp deltas-of-deltas are 0 and most XORs have few changed bits. It's lossless.
Q2/4 A developer adds a user_id label to a request counter on a 10M-user service. What fails first?
Answer & explanation
Answer: B
Each label combination is a separate series with an in-memory head and index entries. Bounded labels and per-metric series limits prevent it.
Q3/4 Why does the alert evaluator get its own Kafka consumer instead of using the dashboard query path?
Answer & explanation
Answer: C
Alerting is the priority. Isolating it means a dashboard overload degrades dashboards, not paging.
Q4/4 How do you find out that the monitoring system itself is down?
Answer & explanation
Answer: D
If the heartbeat stops arriving, an outside service pages. Nothing inside a failed system can be trusted to report its own failure.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.