Every product ends up sending notifications: "your OTP is 482913", "your order has shipped", "50% off this weekend". Building the first one is easy: call the push provider from the order service. The interview question starts when there are fifty services doing that, three channels, users who opted out, providers that go down, and a marketing campaign to 50 million people that must not delay a single login code.
Design a central notification service for a large consumer app. Services such as orders, payments and marketing call it; it decides how to reach the user and delivers through third-party providers (Apple's APNs and Google's FCM for push, an email provider, an SMS gateway).
In scope
- The send API, user preferences, templates
- Reliable delivery through unreliable providers
- Priorities, rate limits and campaigns
One notify API lets every service reach users by push, email or SMS with templates, per-category preferences and quiet hours. Transactional traffic averages an assumed ~116 a second, while a 50M-user campaign runs at ~13,900 a second for an hour and an SMS account limit of 1,000 a second stretches 2.5M SMS to about 42 minutes. The API stores each notification with an outbox and answers 202; Kafka high and bulk topics feed a router that applies preferences and caps, and per-channel workers call APNs, FCM, an email provider and SMS gateways. The deep dives make duplicates harmless with idempotency and per-channel dedupe keys, fail over between providers behind circuit breakers, and keep OTPs in their own lane with reserved provider capacity during campaigns.
Phase 1: Requirements (3 minutes)#
"Which notifications are urgent? Can users opt out per channel and per category? Do we need delivery status? Is in-app inbox in scope?"
"OTPs and payment alerts are urgent. Users choose channels per category, except security messages, which always go out. Callers need a status. Leave the in-app inbox for later."
Functional requirements#
send(user, template, data, priority, idempotency_key)from any internal service- Channels: push (iOS, Android), email, SMS; the system picks based on the request and the user's preferences
- Templates with variables, per channel and language
- Preferences per user, per category, per channel; quiet hours in the user's time zone
- Campaigns: send a template to a segment of users, now or scheduled
- Status per notification and channel: queued, sent, delivered, failed
Non-functional requirements#
- Fast for urgent messages: an OTP reaches the provider within a few seconds, even during a campaign
- No duplicates: a retried API call or a crashed worker must not send the same SMS twice (users notice, and SMS costs money)
- No losses: an accepted notification is delivered or reported as failed, never silently dropped
- Scale: bursts of tens of thousands of messages a second during campaigns
- Resilient: one provider's outage must not stop a whole channel
Phase 2: Estimation (3 minutes)#
- Transactional: assume 10M a day ÷ 86,400 s ≈ ~116/s average, a few hundred a second at peaks
- Campaign: 50M messages in one hour ≈ ~13,900/s for that hour, over 100x the transactional rate
- Channel mix: assume 70% push, 25% email, 5% SMS
- Log: one record per notification per channel attempt, ~1 KB; about 60M a day → ~60 GB/day, ~5.4 TB for 90 days of history
- Providers have their own limits: assume our SMS account allows 1,000 messages/s, so 5% of a 50M campaign (2.5M SMS) takes about 42 minutes no matter how fast we are
"The average is tiny. The design is shaped by the spike: 13,900 a second of low-priority traffic sharing pipes with OTPs that must leave in seconds."
Phase 3: API (2 minutes)#
POST /v1/notifications
Idempotency-Key: order-8812-shipped
{ "user_id": "u_42", "template": "order_shipped", "category": "orders",
"data": { "order_id": "8812", "eta": "Tue" }, "priority": "normal", "channels": ["push", "email"] }
→ 202 Accepted { "notification_id": "n_7f3a" }
GET /v1/notifications/n_7f3a
→ 200 { "push": "delivered", "email": "sent" }
PUT /v1/users/u_42/preferences
{ "orders": { "push": true, "email": false }, "marketing": { "push": false }, "quiet_hours": "22:00-08:00" }
POST /v1/campaigns
{ "segment": "active_last_30d", "template": "weekend_sale", "send_at": "2026-10-31T10:00:00+05:30" }202 Accepted matters: the API promises the notification is stored and will be processed, not that it was delivered.
Phase 4: High-level design (4 minutes)#
%% title: Notification pipeline
flowchart TD
SVC@{ icon: "sd:servers", label: "Internal services<br/>orders, payments, marketing" } --> API["Notify API<br/>validate, idempotency, store + outbox"]:::core
API --> DB@{ icon: "tech:postgresql", label: "PostgreSQL<br/>notifications, status" }
API -->|"outbox relay"| K@{ icon: "tech:kafka", label: "Kafka<br/>high and bulk topics" }
CAMP@{ icon: "sd:cron", label: "Campaign scheduler<br/>segments → batches" } --> K
K --> RT["Router<br/>preferences, quiet hours,<br/>frequency caps, templates"]:::core
RT --> PREF@{ icon: "tech:redis", label: "Redis<br/>preferences, tokens, dedupe keys" }
RT --> PW["Push workers"]:::service
RT --> EW["Email workers"]:::service
RT --> SW["SMS workers"]:::service
PW --> APNS@{ icon: "tech:apple", label: "APNs" }
PW --> FCM@{ icon: "tech:firebase", label: "FCM" }
EW --> SG@{ icon: "tech:sendgrid", label: "SendGrid" }
SW --> SMS@{ icon: "sd:sms", label: "SMS gateways<br/>primary + backup" }
class SVC client
class DB store
class K queue
class CAMP service
class PREF cache
class APNS,FCM,SG,SMS external- Notify API: validates the request, checks the idempotency key, writes the notification row and an outbox entry in one transaction, and returns 202. A relay publishes outbox entries to Kafka, so a notification is never stored without being queued, or queued without being stored.
- Kafka topics by priority:
high(OTP, payments, security) andbulk(marketing, digests), each with its own consumers, so the bulk backlog never sits in front of an OTP. - Router: loads preferences, drops opted-out channels (never for security messages), delays messages that fall in quiet hours, enforces frequency caps, renders the template and emits one job per channel.
- Channel workers: one pool per channel, each calling its providers with retries, rate limits and circuit breakers, and recording the result.
- Campaign scheduler: expands a segment into batches of user IDs at a controlled rate onto the bulk topic.
Phase 5: Deep dives (5 minutes)#
Deep dive 1: Exactly once is impossible, so make duplicates harmless#
"A worker sends an SMS, then crashes before recording success. Kafka redelivers. Does the user get two SMS?"
"Without care, yes. Between our system and an external provider, the best we can do is at-least-once delivery plus deduplication."
- At the API: the caller's
Idempotency-Keyis stored with the notification. A retried POST with the same key returns the originalnotification_idinstead of creating a second one. - At the worker: before calling the provider, the worker sets a dedupe key
(notification_id, channel)in Redis withSET NXand a TTL. If it already exists, the job was handled. This narrows the duplicate window to a crash between the provider call and the status write. - Retry only what can succeed: timeouts, 5xx and 429 are retried with exponential backoff and jitter. Permanent errors are not: an invalid phone number fails at once, and a push token that APNs or FCM reports as no longer valid is deactivated.
- Give up visibly: after a set number of attempts, the job goes to a dead-letter queue and the status becomes
failed, so the caller can see it and act.
A circuit breaker per provider stops sending to one that is failing, instead of piling up timeouts. For SMS and email, keep a second provider and fail over when the breaker opens. Push has no alternative for a given platform (APNs is the only way to reach an iPhone), so push jobs wait and retry.
Deep dive 2: Keeping urgent messages fast during a 50M campaign#
"Marketing launches a 50 million user campaign at 10 AM. A minute later someone requests a login OTP. What does your design do?"
- Separate topics and consumers: the OTP goes to
high, the campaign tobulk. They don't share partitions, consumer groups or worker pools. - Separate provider budgets: split each provider's rate limit with token buckets, for example reserving 30% of SMS throughput for
higheven when bulk would use all of it. - Throttle the campaign at the source: the scheduler releases batches at a rate the providers can take (with 700 SMS a second left for bulk after the reservation, 2.5M SMS take about an hour), rather than dumping 50M jobs at once.
- Watch the right metric: alert on the age of the oldest message in
high, not on total queue depth, which a campaign makes huge on purpose.
- Frequency caps: at most, say, 3 marketing pushes per user per day, checked in the router
- Quiet hours: a marketing message that would arrive at 2 AM in the user's time zone is scheduled for 8 AM; security messages ignore quiet hours
- Collapse: five "new comment" notifications in a minute become one digest
Duplicates and priorities
Q1/2 A worker sends an SMS, crashes before saving the status, and Kafka redelivers the job. What prevents a second SMS?
Answer & explanation
Answer: B
At-least-once delivery plus a
(notification_id, channel)dedupe key makes the redelivered job a no-op. Kafka alone can't make an external call exactly-once.Q2/2 During a 50M-message campaign, which metric best shows whether OTPs are at risk?
Answer & explanation
Answer: C
A campaign makes total depth large on purpose. What matters for OTPs is how long the oldest urgent message has waited.
Phase 6: Trade-offs and failure modes (2 minutes)#
Kafka or the router falls behind
The API keeps accepting (202) because notifications are stored with an outbox; delivery resumes when consumers catch up. Shed bulk work first: pause the campaign scheduler before touching high.
Order of related messages
"Delivered" can overtake "shipped" when they travel through different retries. Partition by user ID so one user's messages stay in order on a topic, and include a sequence so the app can drop an older status that arrives late.
Stale device tokens
Users reinstall apps and switch phones. Remove tokens the providers report as invalid, and re-register tokens on every app start.
Preferences change mid-flight
Check preferences at routing time, not at API time, so an opt-out takes effect for messages still queued. Security messages bypass opt-out by design.
Cost
SMS is the expensive channel. Prefer push when the user has a working token, use SMS for OTPs and fallbacks, and alert on SMS spend per hour.
Phase 7: Go deeper in the course#
- This whole system, day by day: Week 6 · Day 1: Notification Platform Design, Day 2: Core Notification Flow, Day 3: Advanced Features, Day 4: Scale, Reliability and Edge Cases and Day 5: Operations and Monitoring
- Stored and queued atomically: Week 3 · Day 2: Transactional Outbox
- Idempotency keys end to end: Week 2 · Day 2: Idempotency in Practice
- Failing providers: Week 2 · Day 3: Circuit Breakers
- Retries done right: Lab: Retry with exponential backoff and full jitter
- Jobs that keep failing: Week 3 · Day 4: Dead Letters and Poison Pills
- Provider rate limits and token buckets: Week 1 · Day 3: Rate Limiting at Scale
- Delivery callbacks from providers: Week 2 · Day 4: Webhook Delivery System
Final check#
Final check · Notification system
Q1/3 50M campaign messages go out in one hour. Roughly what rate is that?
Answer & explanation
Answer: B
50,000,000 ÷ 3,600 s ≈ 13,900 a second, over 100 times the ~116 a second transactional average.
Q2/3 Why does the API answer 202 Accepted rather than 200 OK?
Answer & explanation
Answer: B
The notification is persisted with an outbox and processed asynchronously. Delivery status is available later through GET.
Q3/3 The primary SMS gateway starts timing out. What should the SMS workers do?
Answer & explanation
Answer: D
A breaker stops the pile-up of timeouts, and a second SMS provider keeps the channel up. Push has no such alternative per platform, so it waits and retries.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.