Designing a Notification Platform
Day 1Notification Platform — Problem Understanding & High-Level Design
Ask before you draw. Spend the first five minutes on questions (FERNS: functional, edge cases, resources, non-functional, scale). Here they reveal five notification types, five channels, preferences, quiet hours, digests, fallback and compliance.
Priority decides everything. Critical security alerts go out on several channels at once with maximum retries; transactions are high priority with a fallback; marketing is low, batchable and delayable. Marketing must never delay transactions.
Every channel behaves differently. Push is free but best-effort, with tokens that rotate. Email is cheap, slow and fights spam filters. SMS is reliable but expensive with a 160-character limit. In-app needs a notification center for offline users.
The pipeline. Send API writes the notification and an outbox row in one transaction; the outbox publishes to priority Kafka topics; a router applies preferences and quiet hours; channel workers render and call providers.
Compliance is a requirement. CAN-SPAM needs a working unsubscribe and opt-outs within 10 business days; GDPR needs explicit marketing consent; TCPA bars marketing SMS outside 8am–9pm local and honours STOP. Keep a full audit trail.
| Push | SMS | ||
|---|---|---|---|
| Cost | Free | $0.10–$1.00 per 1,000 | $0.01–$0.10 each |
| Delivery | Best effort | Reliable, may hit spam | High |
| Use for | Primary channel | Fallback, digests | Critical alerts, OTP |
- Drawing boxes before clarifying scale, latency and compliance.
- Sending marketing through the same queue as security alerts.
- Using SMS as a default channel: push first, email second, SMS last.
Day 2Core Notification Flow
The Send API returns fast. Validate, assign a notification ID, save it with an outbox row in one transaction and return the ID. Delivery happens asynchronously, idempotency keys make retries safe, and bad requests never reach the queue.
Why the outbox here. Save-then-publish can save a notification that's never processed (the user is never told) or process one that was never saved (no status, possible duplicates). The outbox makes the save and the event atomic.
Route by preference. The router consumes from Kafka, reads the user's preferences (Redis first, then PostgreSQL), checks quiet hours and channel availability, and publishes one task per channel topic.
One interface for every provider. FCM, APNs, SendGrid and Twilio sit behind the same provider interface. Channel workers consume their own topic, retry with exponential backoff and send exhausted messages to a dead letter queue.
Status is a state machine. PENDING, QUEUED, ROUTING, SENDING, then SENT, FAILED or BOUNCED; SENT can still become DELIVERED or BOUNCED when provider webhooks arrive later, then READ. FAILED goes back to QUEUED for a retry.
| Save, then publish | Outbox | |
|---|---|---|
| Kafka down | Saved, never sent | Waits in the outbox |
| Database fails | Sent, not tracked | Nothing written, nothing sent |
| Result | Lost or duplicate notifications | Never lost |
- Waiting for delivery inside the Send API instead of returning the ID.
- Letting invalid payloads into the queue, where they fail late.
- Treating a provider's "accepted" response as delivered.
Day 3Advanced Features
Batch what would be noise. 50 trades, 20 likes or 10 transactions shouldn't be 50, 20 or 10 notifications. Batch by count, by time window or both ("20 people liked your post") with a worker that flushes batches.
Digests and scheduling. Digests are timezone-aware daily, weekly or monthly summaries configured per user. Scheduled and recurring (cron) notifications convert to the user's timezone, can be cancelled, and the scheduler runs only on the elected leader.
A/B test templates. Templates carry variants with weights. Assign each user consistently to one variant, measure engagement and check statistical significance before declaring a winner.
Multi-channel delivery is a saga. Critical alerts go out on primary channels together, track each channel's state, fall back on failure and escalate: push first, SMS if unopened after 5 minutes, a phone call at the 15-minute check.
The in-app notification center. Store notifications with read/unread state and grouping, push them over a WebSocket gateway when the user is online. Preferences are hierarchical: global, then category, then channel, with quiet hours.
| Immediate | Batched or digest | |
|---|---|---|
| Latency | Seconds | Window or schedule |
| User noise | High for bursts | One summary |
| Fits | Security, transactions | Likes, trades, marketing |
- Sending every like or trade as its own push notification.
- Scheduling in server time instead of each user's timezone.
- Updating preferences without a version check: optimistic locking prevents lost updates.
Day 4Scale, Reliability & Edge Cases
Campaigns arrive as a wall. 10M in an hour is only 2,800/s on top of ~5,800/s, but marketing clicks Send once: 100,000+/s for the first seconds. Rate-limit campaign ingestion, batch it and isolate it on low-priority topics and workers.
Hot users fan out. A celebrity post to 1M followers means 1M preference and token lookups at once on a hot partition. Fan out asynchronously in batches, query followers in batches and route partition-aware.
Respect provider limits, survive provider failure. Per-provider sliding-window rate limiters with backoff on rate-limit errors; a circuit breaker per provider; automatic failover (SendGrid to SES, Twilio to SNS); degraded modes that shed load.
The edge case encyclopedia. Rotating and stale device tokens, multiple devices, hard vs soft email bounces and spam complaints, quiet hours across DST, and users who change preferences or are deleted mid-delivery.
Re-check at delivery time. A user changed their email while a notification was queued for the old one. Read preferences and contact details again when delivering, not only when queueing.
| Impact | Recovery | |
|---|---|---|
| PostgreSQL primary | Writes fail | Promote a replica |
| Redis | Cache misses | Fall back to the DB |
| All of Kafka | Can't queue | Buffer in memory or disk |
| FCM / SendGrid | Delivery fails | Circuit break, failover |
- Letting a campaign burst share workers with transaction notifications.
- Generating 1M notification requests synchronously for one celebrity post.
- Keeping stale device tokens and bounced addresses: they waste sends and hurt reputation.
Day 5Operations, Monitoring & Interview Mastery
Three pillars of observability. Metrics for what's happening now (delivery rate, queue depth, provider latency, circuit states), structured logs, and distributed traces that follow one notification from API to router, worker, provider and callback.
Alerts that don't cry wolf. Page when delivery drops below 95%, warn when a circuit opens, investigate when queue lag passes 1M. Every alert has a runbook: low delivery rate, Kafka backlog, provider circuit open.
Separate ingestion from delivery. API, Kafka, router, channel workers, providers, then callbacks. Ingestion returns immediately; Kafka buffers and enables retry; delivery scales and fails on its own.
The 45-minute framework. Requirements 0–5 min, estimation 5–10, high-level design 10–25, two or three deep dives 25–40, wrap-up 40–45. Cover multi-channel delivery, reliability, scale, preferences and observability.
Lessons from the giants. Uber: P0/P1/P2 priority classes on separate infrastructure. Slack: online users get WebSocket, offline get push after a delay. WhatsApp: push only says "check for messages", content stays end-to-end encrypted.
| Prevents | How | |
|---|---|---|
| Outbox | Lost notifications | DB + outbox in one transaction |
| Idempotency key | Duplicates | Checked in Redis first |
| Commit after success | Silent drops | At-least-once + idempotency |
| DLQ | Endless retries | Park after 3 failures |
- No traces, so a slow notification can't be pinned to the router, worker or provider.
- Alerts without runbooks, or so many that on-call ignores them.
- Committing Kafka offsets before the provider call succeeds.
BossFood Delivery Order System
Size the dinner rush. 5M orders a day is ~58 a second; dinner is 4x, ~230 a second. Each order brings status updates, location points and notifications: ~7,000 writes and ~60,000 reads a second at peak.
Never lose an order. Write the order and an outbox row in one transaction; the publisher marks rows published only after Kafka confirms, so a crash just republishes, and idempotent consumers absorb the duplicate.
Payments: authorize, capture, refund. Authorize at order time, capture when the driver picks up, refund on cancellation, every Stripe call with an idempotency key. Without one, a retried authorization after a timeout creates a second PaymentIntent.
No driver assigned twice. Two orders can both see Driver X as free and both take them. Assign atomically with row locking and a version check, so only one assignment succeeds and the other picks another driver.
The order saga. Restaurant rejects after payment is authorized: void the authorization and notify the customer. Payment capture fails after pickup: the food is gone, so retry with backoff, then flag for manual review without blocking delivery.
| First | Later | |
|---|---|---|
| Orders | Outbox order flow | Full saga orchestration |
| Payments | Idempotent integration | Backup processor |
| Drivers | Nearest available | Optimal matching |
| Tracking | Status updates | Live ETAs |
- Saving the order, then publishing to Kafka: Kafka down means a charged customer and no food.
- Retrying payment calls without an idempotency key.
- Blocking delivery because payment capture failed after pickup.