Bonus Problem 2: WhatsApp Messaging
How 50 Engineers Built the World's Most Efficient Messaging Platform
📱 The Impossible Made Simple
When Facebook acquired WhatsApp for $19 billion in 2014, something didn't add up.
The messaging app was handling 50 billion messages daily — but with only 32 engineers on the backend.
How is that even possible?
THE WHATSAPP PARADOX (2014 Acquisition)
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ MESSAGES PER DAY ENGINEERS │
│ ────────────────── ───────── │
│ 50 Billion 32 (backend) │
│ 50 (total) │
│ │
│ SERVERS COST PER USER │
│ ─────── ───────────── │
│ ~550 Fraction of competitors │
│ │
│ CONNECTIONS PER SERVER TECHNOLOGY STACK │
│ ────────────────────── ──────────────── │
│ 2-3 Million Erlang + FreeBSD │
│ (Industry: ~100K typical) (Not Java, not Linux) │
│ │
│ For comparison: │
│ • Facebook Messenger: 1000+ engineers │
│ • Most messaging apps: 100-500 engineers for similar scale │
│ │
│ WhatsApp achieved 20x efficiency. How? │
│ │
└──────────────────────────────────────────────────────────────────────────┘
TODAY'S SCALE (2025)
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ MONTHLY ACTIVE USERS DAILY MESSAGES │
│ ──────────────────── ────────────── │
│ 2.5+ Billion 100+ Billion │
│ │
│ PEAK MESSAGES/SECOND COUNTRIES │
│ ─────────────────── ───────── │
│ 3+ Million 180+ │
│ │
│ STATUS UPDATES/DAY VOICE/VIDEO CALLS │
│ ────────────────── ───────────────── │
│ 500+ Million 2+ Billion minutes/day │
│ │
│ ENCRYPTION UPTIME │
│ ────────── ────── │
│ End-to-End (Signal Protocol) 99.99%+ │
│ WhatsApp can't read messages │
│ │
└──────────────────────────────────────────────────────────────────────────┘
This is the system we'll design today — and understand why their technology choices matter.
The Interview Begins
You're interviewing at a startup that wants to build the next great messaging app. The CTO draws on the whiteboard:
Interviewer: "WhatsApp proved that a small team can build a messaging platform for billions. I want you to design a messaging system that can scale to that level. Walk me through how you'd approach it."
╔══════════════════════════════════════════════════════════════════════════╗
║ ║
║ Design a Global Messaging Platform ║
║ ║
║ Build a messaging system that can handle billions of users with: ║
║ ║
║ Requirements: ║
║ • Support 2 billion users ║
║ • Handle 100+ billion messages per day ║
║ • Real-time message delivery (< 500ms for online users) ║
║ • Offline message storage and delivery ║
║ • End-to-end encryption (server cannot read messages) ║
║ • Group messaging (up to 1000 members) ║
║ • Media sharing (images, video, documents) ║
║ • Presence (online/offline/typing indicators) ║
║ • Message status (sent, delivered, read) ║
║ • 99.99% availability ║
║ ║
║ Constraint: Build it with a small team (< 100 engineers) ║
║ ║
╚══════════════════════════════════════════════════════════════════════════╝
Interviewer: "The team size constraint is intentional. I want to see if you understand that architecture choices determine how many engineers you need. WhatsApp proved you can do more with less."
Phase 1: Requirements Clarification
You: "Let me make sure I understand the constraints and can make the right trade-offs."
Your Questions
You: "First, what's the message delivery guarantee? Is it okay if a message is delayed, or must it be truly real-time?"
Interviewer: "For online users, we want sub-second delivery. For offline users, messages should be queued and delivered when they come online. Messages should never be lost."
You: "What about message ordering? If I send two messages quickly, must they arrive in order?"
Interviewer: "Within a single conversation, yes — messages must be ordered. But I don't care if Message A to Bob arrives before Message B to Alice, even if B was sent first."
You: "For end-to-end encryption, does the server ever need to read message content?"
Interviewer: "Never. The server should only route encrypted blobs. This is non-negotiable for user trust."
You: "What's the read/write ratio? How often do users check for messages vs send them?"
Interviewer: "Heavy read. Users open the app frequently to check for messages. For every message sent, there are probably 10 message reads/checks. Also, presence updates are extremely chatty — typing indicators, online status."
You: "What about media? Video messages can be large."
Interviewer: "Media follows a different path. Upload to storage, send a reference in the message. The encrypted media is stored separately from chat servers."
Requirements Summary
Functional Requirements:
1. MESSAGING
• One-to-one messaging
• Group messaging (up to 1024 members)
• Message types: text, image, video, audio, document, location
• Message status: sent → delivered → read
• Reply, forward, delete (for everyone)
2. PRESENCE
• Online/offline status
• Last seen timestamp
• Typing indicators
• Privacy controls (hide last seen)
3. MEDIA HANDLING
• Upload and download media
• Automatic compression
• Progressive loading (blurry → clear)
• Media forwarding (reuse uploaded media)
4. SECURITY
• End-to-end encryption for all messages
• Device verification (QR code scan)
• Two-factor authentication
5. SYNC
• Multi-device support
• Message sync across devices
• Local backup (encrypted)
Non-Functional Requirements:
SCALE
• 2 billion users
• 500 million daily active users
• 100 billion messages/day
• 3 million messages/second (peak)
LATENCY
• Online delivery: < 500ms
• Offline delivery: < 5s after coming online
• Typing indicators: < 200ms
AVAILABILITY
• 99.99% uptime
• Graceful degradation (messages queue if delays)
STORAGE
• Messages stored until delivered
• Delivered messages deleted from server
• Media retained for 30 days
EFFICIENCY
• Minimal bandwidth (works on 2G)
• Minimal battery drain
• Small engineering team
Phase 2: Back of the Envelope Estimation
You: "Let me work through the numbers to understand infrastructure needs."
Traffic Calculations
MESSAGE VOLUME
Daily messages: 100,000,000,000 (100 billion)
Seconds per day: 86,400
Average MPS: ~1.16 million messages/second
Peak multiplier: 3x (evening hours globally)
Peak MPS: ~3.5 million messages/second
Each message triggers:
├── Sender acknowledgment: 1 message
├── Recipient delivery: 1 message
├── Read receipt: 1 message
└── Total events/message: ~3-4x
Effective events/second: ~10-15 million at peak
Connection Calculations
CONCURRENT CONNECTIONS
Daily active users: 500 million
Peak concurrent: ~10% = 50 million connections
If each server handles 2 million connections:
├── Servers needed: 25 servers (just for connections!)
├── With redundancy: 50-75 servers
WhatsApp achieved 2-3 million connections per server.
Most companies: 50,000-100,000 per server.
This 20-40x efficiency is the key insight.
Storage Calculations
MESSAGE STORAGE
Average message size: ~500 bytes (encrypted)
Daily messages: 100 billion
Daily storage: 50 TB/day (if we stored everything)
BUT: WhatsApp deletes messages after delivery!
├── Undelivered at any time: ~5% of daily messages
├── Actual storage: ~2.5 TB active
└── Plus offline queues: ~10 TB
Media storage is separate:
├── Daily uploads: ~10 billion media files
├── Average size: 100 KB (compressed)
├── Daily media: ~1 PB/day
├── 30-day retention: ~30 PB
└── With deduplication: ~15 PB
Phase 3: High-Level Architecture
You: "WhatsApp's architecture is remarkably simple. That's the secret — simplicity enables a small team."
The WhatsApp Philosophy
WHATSAPP'S ENGINEERING PRINCIPLES
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ 1. FOCUS │
│ One thing, done exceptionally well: messaging. │
│ No social feed, no stories (initially), no ads. │
│ Feature creep is the enemy of reliability. │
│ │
│ 2. RELIABILITY OVER FEATURES │
│ A message that doesn't deliver is worse than no feature. │
│ Every new feature adds complexity and potential failures. │
│ Say no to most feature requests. │
│ │
│ 3. RIGHT TOOL FOR THE JOB │
│ Erlang for concurrency (designed for telecom switches). │
│ FreeBSD for networking performance. │
│ Mnesia for in-memory distributed data. │
│ Unconventional, but perfect fit. │
│ │
│ 4. SMALL TEAM = BETTER COMMUNICATION │
│ 32 engineers means everyone knows the whole system. │
│ Every line of code reviewed by founders (initially). │
│ No coordination overhead between 50 microservice teams. │
│ │
│ 5. LET IT CRASH │
│ Erlang philosophy: processes crash, supervisors restart. │
│ Don't try to handle every error — let it fail and recover. │
│ Simple, predictable failure recovery. │
│ │
└─────────────────────────────────────────────────────────────────────────┘
System Architecture
WHATSAPP ARCHITECTURE
┌────────────────────────────────────────────────────────────────────────┐
│ │
│ CLIENT LAYER │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Android │ │ iOS │ │ WhatsApp │ │
│ │ App │ │ App │ │ Web │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ └────────────────┼────────────────┘ │
│ │ │
│ │ (Persistent TCP + TLS) │
│ │ (Custom binary protocol) │
│ │ │
└────────────────────────────┼───────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ EDGE / GATEWAY │
│ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ LOAD BALANCERS │ │
│ │ (GeoDNS routes to nearest data center) │ │
│ └──────────────────────────┬──────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ CHAT SERVERS │ │
│ │ (Erlang + FreeBSD) │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │ Chat 1 │ │ Chat 2 │ │ Chat 3 │ │ Chat N │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ │ 2-3M │ │ 2-3M │ │ 2-3M │ │ 2-3M │ │ │
│ │ │ conns │ │ conns │ │ conns │ │ conns │ │ │
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │
│ │ │ │
│ │ Each connection = 1 Erlang process (~2KB memory) │ │
│ │ Process handles: auth, presence, message routing │ │
│ │ │ │
│ └──────────────────────────┬──────────────────────────────────────┘ │
│ │ │
└───────────────────────────────┼─────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ │
│ BACKEND SERVICES │
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ MESSAGE │ │ OFFLINE │ │ MEDIA │ │
│ │ ROUTING │ │ STORE │ │ STORAGE │ │
│ │ │ │ │ │ │ │
│ │ Direct P2P │ │ Queue for │ │ S3-like │ │
│ │ between │ │ offline │ │ blob store │ │
│ │ chat servers │ │ users │ │ for media │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ USER │ │ GROUP │ │ KEYS │ │
│ │ REGISTRY │ │ METADATA │ │ (for E2E) │ │
│ │ │ │ │ │ │ │
│ │ User→Server │ │ Group │ │ Public key │ │
│ │ mapping │ │ membership │ │ distribution │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │
│ All backed by: Mnesia (Erlang distributed DB) + ETS (in-memory) │
│ │
└────────────────────────────────────────────────────────────────────────┘
Message Flow
MESSAGE FLOW: ALICE SENDS "HELLO" TO BOB
Alice's Phone WhatsApp Servers Bob's Phone
│ │ │
│ ① ENCRYPT │ │
│ Message encrypted │ │
│ with Bob's public key │ │
│ (Signal Protocol) │ │
│ │ │
│ ② SEND │ │
│ ─────────────────────────▶ │
│ Encrypted blob + │ │
│ recipient: bob@wa │ │
│ │ │
│ ③ ACK (SENT ✓) │ │
│ ◀───────────────────────── │
│ Server received message │ │
│ (Single tick) │ │
│ │ │
│ │ ④ LOOKUP │
│ │ Where is Bob connected? │
│ │ User registry → Server 7 │
│ │ │
│ │ │
│ IF BOB IS ONLINE │
│ │ │
│ │ ⑤ ROUTE │
│ │ ─────────────────────────▶│
│ │ Forward to Bob's process │
│ │ │
│ │ ⑥ DELIVERED ACK │
│ │ ◀─────────────────────────│
│ │ Bob's device got it │
│ │ │
│ ⑦ DELIVERED (✓✓) │ │
│ ◀───────────────────────── │
│ (Double tick) │ │
│ │ │
│ │ │
│ BOB READS THE MESSAGE │
│ │ │
│ │ ⑧ READ RECEIPT │
│ │ ◀─────────────────────────│
│ │ Bob opened the chat │
│ │ │
│ ⑨ READ (✓✓ blue) │ │
│ ◀───────────────────────── │
│ (Blue double tick) │ │
│ │ │
▼ ▼ ▼
KEY INSIGHT: Server NEVER decrypts the message.
It only routes the encrypted blob and tracks delivery status.
Phase 4: Deep Dives
Deep Dive 1: Why Erlang?
Week 2 concepts: Failure handling, resilience. Week 10 concepts: Operational simplicity.
You: "WhatsApp's secret weapon is Erlang — a language designed in the 1980s for telephone switches. It's a perfect fit for messaging."
The Challenge:
THE CONCURRENCY PROBLEM
Traditional approach (Java/Python/Go):
├── Each connection = 1 OS thread
├── OS thread = ~1MB memory overhead
├── Context switching is expensive
├── 1 million connections = 1 million threads = 1TB RAM!
└── Practical limit: ~50,000-100,000 connections/server
WhatsApp needs:
├── 2-3 million connections per server
├── Each connection needs state (session, queue)
├── Must handle failures gracefully
└── Hot code reloading (no restarts for deploys)
Why Erlang Solves This:
ERLANG'S SUPERPOWERS
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ 1. LIGHTWEIGHT PROCESSES │
│ ───────────────────────── │
│ │
│ Erlang processes are NOT OS threads. │
│ They're managed by the BEAM virtual machine. │
│ │
│ OS Thread: Erlang Process: │
│ • ~1MB memory • ~2KB memory │
│ • Expensive switch • Cheap switch │
│ • Limited count • Millions possible │
│ │
│ WhatsApp runs 1 Erlang process per user connection. │
│ 2 million connections = 2 million processes = ~4GB RAM │
│ (vs 2TB with OS threads) │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 2. "LET IT CRASH" PHILOSOPHY │
│ ──────────────────────────── │
│ │
│ Traditional: Try to handle every error (complex, error-prone) │
│ Erlang: Let processes crash, supervisors restart them │
│ │
│ ┌─────────────────┐ │
│ │ SUPERVISOR │ (monitors children) │
│ └────────┬────────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────┐ ┌───────┐ ┌───────┐ │
│ │User A │ │User B │ │User C │ │
│ │Process│ │Process│ │Process│ │
│ └───────┘ └───────┘ └───────┘ │
│ │ │
│ │ CRASH! │
│ ▼ │
│ ┌───────┐ │
│ │User B │ ← Supervisor restarts │
│ │Process│ in milliseconds │
│ └───────┘ │
│ │
│ User A and C never notice. User B reconnects automatically. │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 3. HOT CODE RELOADING │
│ ───────────────────── │
│ │
│ Deploy new code WITHOUT restarting the server. │
│ Connections stay alive during deployments. │
│ No maintenance windows needed. │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 4. DISTRIBUTED BY DESIGN │
│ ───────────────────────── │
│ │
│ Erlang processes can communicate across machines seamlessly. │
│ Send a message to a process on another server = same syntax. │
│ Built-in distribution, no external message queue needed. │
│ │
└─────────────────────────────────────────────────────────────────────────┘
WhatsApp's Erlang Customizations:
# Pseudocode: How WhatsApp structures connection handling
# (Actual WhatsApp code is Erlang, this shows the concept)
"""
WhatsApp Connection Handler
Each user connection runs as a single Erlang process.
This process:
1. Owns the TCP socket
2. Maintains user session state
3. Routes messages to/from the user
4. Dies cleanly when user disconnects
"""
class UserConnectionProcess:
"""
Conceptual model of WhatsApp's per-user Erlang process.
In Erlang, this is implemented with gen_server behavior.
Memory: ~2KB per process
Lifecycle: Lives as long as user is connected
"""
def __init__(self, socket, user_id):
self.socket = socket
self.user_id = user_id
self.session_key = None
self.message_queue = [] # For messages while processing
self.last_seen = now()
# Register this process in the global user registry
UserRegistry.register(user_id, self.pid)
def handle_message(self, msg):
"""
Main message handler — called for every incoming message.
Erlang processes have a mailbox — messages queue up
and are processed one at a time.
"""
match msg:
case AuthRequest(credentials):
self.handle_auth(credentials)
case OutgoingMessage(recipient, encrypted_content):
self.route_to_recipient(recipient, encrypted_content)
case IncomingMessage(sender, encrypted_content):
self.deliver_to_socket(sender, encrypted_content)
case PresenceUpdate(status):
self.broadcast_presence(status)
case Disconnect():
self.cleanup_and_die()
def route_to_recipient(self, recipient_id, content):
"""
Route message to another user.
This is the core of WhatsApp's routing:
1. Look up recipient's process (which server?)
2. Send message directly to that process
"""
# Find recipient's process (could be on any server)
recipient_process = UserRegistry.lookup(recipient_id)
if recipient_process:
# Direct process-to-process communication
# Works across servers in Erlang!
send(recipient_process, IncomingMessage(
sender=self.user_id,
content=content
))
self.send_ack(MessageStatus.DELIVERED)
else:
# User offline — store for later
OfflineStore.queue(recipient_id, content)
self.send_ack(MessageStatus.SENT)
def cleanup_and_die(self):
"""
Clean shutdown when user disconnects.
In Erlang, when a process dies, all its memory
is automatically reclaimed. No memory leaks possible.
"""
UserRegistry.unregister(self.user_id)
close(self.socket)
# Process simply terminates — Erlang handles cleanup
The "Island" Architecture:
WHATSAPP'S ISLAND ARCHITECTURE
To limit blast radius of failures, WhatsApp partitions
servers into independent "islands."
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ ISLAND 1 ISLAND 2 ISLAND 3 │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ Chat Servers │ │ Chat Servers │ │ Chat Servers │ │
│ │ 1, 2, 3, 4 │ │ 5, 6, 7, 8 │ │ 9, 10, 11 │ │
│ │ │ │ │ │ │ │
│ │ Users A-H │ │ Users I-P │ │ Users Q-Z │ │
│ │ │ │ │ │ │ │
│ │ Replica: 1a │ │ Replica: 2a │ │ Replica: 3a │ │
│ └─────────────────┘ └─────────────────┘ └──────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ └──────────────────────────┴─────────────────────────┘ │
│ │ │
│ ▼ │
│ Cross-Island Routing │
│ (for A→Z messages) │
│ │
│ BENEFITS: │
│ • Island 1 can fail without affecting Island 2 │
│ • Upgrades can be rolled out island by island │
│ • Each island has its own replica for failover │
│ • Limits "blast radius" of any failure │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Deep Dive 2: End-to-End Encryption (Signal Protocol)
Week 9 concepts: Security, encryption.
You: "WhatsApp uses the Signal Protocol for end-to-end encryption. This is critical — even WhatsApp can't read your messages."
END-TO-END ENCRYPTION OVERVIEW
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ WITHOUT E2EE: │
│ ───────────── │
│ │
│ Alice ──"Hi"──▶ Server ──"Hi"──▶ Bob │
│ │ │
│ │ Server can read "Hi" │
│ │ Hackers who breach server can read "Hi" │
│ │ Government subpoenas can access "Hi" │
│ │
│ WITH E2EE: │
│ ────────── │
│ │
│ Alice ──[encrypted blob]──▶ Server ──[encrypted blob]──▶ Bob │
│ │ │
│ │ Server sees random bytes │
│ │ Only Alice and Bob can decrypt │
│ │ Even WhatsApp can't read it │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Signal Protocol Components:
SIGNAL PROTOCOL: THE BUILDING BLOCKS
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ 1. KEY PAIRS │
│ ───────────── │
│ │
│ Each user has multiple key pairs: │
│ │
│ Identity Key (long-term): │
│ • Generated once when you install WhatsApp │
│ • Never changes (unless you reinstall) │
│ • Public part uploaded to WhatsApp servers │
│ │
│ Signed Pre-Key (medium-term): │
│ • Rotates periodically │
│ • Signed by Identity Key │
│ • Uploaded to servers │
│ │
│ One-Time Pre-Keys (ephemeral): │
│ • Batch of ~100 keys uploaded │
│ • Each used once then discarded │
│ • Provides forward secrecy │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 2. X3DH KEY AGREEMENT │
│ ───────────────────── │
│ │
│ When Alice messages Bob for the first time: │
│ │
│ Alice Server Bob │
│ │ │ │ │
│ │ "Give me Bob's keys" │ │ │
│ │────────────────────────────▶│ │ │
│ │ │ │ │
│ │ Bob's: Identity Key │ │ │
│ │ Signed Pre-Key │ │ │
│ │ One-Time Pre-Key │ │ │
│ │◀────────────────────────────│ │ │
│ │ │ │ │
│ │ │ │
│ │ Alice computes shared secret using: │ │
│ │ • Her Identity Key + Bob's Signed Pre-Key │ │
│ │ • Her Identity Key + Bob's One-Time Pre-Key │ │
│ │ • Her Ephemeral Key + Bob's Identity Key │ │
│ │ • Her Ephemeral Key + Bob's Signed Pre-Key │ │
│ │ │ │
│ │ → Shared Secret (only Alice and Bob can compute) │ │
│ │ │ │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 3. DOUBLE RATCHET │
│ ───────────────── │
│ │
│ After initial key agreement, keys evolve with EVERY message: │
│ │
│ Message 1: Encrypt with Key_1 │
│ Message 2: Encrypt with Key_2 (derived from Key_1) │
│ Message 3: Encrypt with Key_3 (derived from Key_2) │
│ ... │
│ │
│ Benefits: │
│ • Forward Secrecy: Compromise of Key_3 doesn't reveal Msg 1-2 │
│ • Post-Compromise Security: If key leaked, future keys are safe │
│ • Break-in Recovery: System "heals" itself over time │
│ │
└─────────────────────────────────────────────────────────────────────────┘
# encryption/signal_protocol.py
"""
Simplified illustration of Signal Protocol concepts.
The actual implementation uses:
- Curve25519 for key exchange
- AES-256-CBC for encryption
- HMAC-SHA256 for authentication
"""
from dataclasses import dataclass
from typing import Optional, Tuple
import os
import hashlib
import hmac
@dataclass
class KeyPair:
"""A public/private key pair."""
public_key: bytes
private_key: bytes
@dataclass
class SessionState:
"""State for an encrypted session with another user."""
their_identity_key: bytes
their_signed_prekey: bytes
# Ratchet state
root_key: bytes
chain_key: bytes
message_number: int = 0
class SignalSession:
"""
Simplified Signal Protocol session.
This illustrates the key concepts — actual implementation
is more complex with proper cryptographic primitives.
"""
def __init__(
self,
my_identity: KeyPair,
their_identity_public: bytes,
their_signed_prekey: bytes,
their_one_time_prekey: Optional[bytes] = None
):
self.my_identity = my_identity
# Perform X3DH key agreement
self.session = self._x3dh_key_agreement(
their_identity_public,
their_signed_prekey,
their_one_time_prekey
)
def _x3dh_key_agreement(
self,
their_identity: bytes,
their_signed_prekey: bytes,
their_one_time_prekey: Optional[bytes]
) -> SessionState:
"""
X3DH: Extended Triple Diffie-Hellman.
Creates a shared secret from multiple DH exchanges.
"""
# Generate ephemeral key for this session
ephemeral = self._generate_keypair()
# Perform multiple DH exchanges
dh1 = self._dh(self.my_identity.private_key, their_signed_prekey)
dh2 = self._dh(ephemeral.private_key, their_identity)
dh3 = self._dh(ephemeral.private_key, their_signed_prekey)
if their_one_time_prekey:
dh4 = self._dh(ephemeral.private_key, their_one_time_prekey)
shared_secret = self._kdf(dh1 + dh2 + dh3 + dh4)
else:
shared_secret = self._kdf(dh1 + dh2 + dh3)
# Derive initial root and chain keys
root_key, chain_key = self._derive_keys(shared_secret)
return SessionState(
their_identity_key=their_identity,
their_signed_prekey=their_signed_prekey,
root_key=root_key,
chain_key=chain_key
)
def encrypt(self, plaintext: bytes) -> Tuple[bytes, int]:
"""
Encrypt a message using current chain key.
Each message gets a unique key (Double Ratchet).
"""
# Derive message key from chain key
message_key, new_chain_key = self._ratchet_chain(
self.session.chain_key
)
# Encrypt with message key
ciphertext = self._aes_encrypt(message_key, plaintext)
# Update chain key (ratchet forward)
message_number = self.session.message_number
self.session.chain_key = new_chain_key
self.session.message_number += 1
return ciphertext, message_number
def decrypt(self, ciphertext: bytes, message_number: int) -> bytes:
"""
Decrypt a message.
May need to skip ahead if messages arrive out of order.
"""
# Derive the message key for this message number
message_key = self._get_message_key(message_number)
# Decrypt
plaintext = self._aes_decrypt(message_key, ciphertext)
return plaintext
def _ratchet_chain(self, chain_key: bytes) -> Tuple[bytes, bytes]:
"""
Symmetric ratchet: derive next keys from current.
This is the "forward secrecy" mechanism — old keys
can't be derived from new keys.
"""
# Message key for current message
message_key = hmac.new(chain_key, b"\x01", hashlib.sha256).digest()
# New chain key for next message
new_chain_key = hmac.new(chain_key, b"\x02", hashlib.sha256).digest()
return message_key, new_chain_key
class GroupEncryption:
"""
Group messaging uses a different approach.
Instead of pairwise encryption, sender distributes
a "sender key" to all group members.
"""
def __init__(self, group_id: str, my_user_id: str):
self.group_id = group_id
self.my_user_id = my_user_id
# Generate my sender key for this group
self.my_sender_key = self._generate_sender_key()
# Store other members' sender keys
self.member_sender_keys = {}
def distribute_sender_key(self, member_sessions: dict):
"""
Send my sender key to all group members.
Uses pairwise Signal sessions for secure distribution.
"""
for member_id, session in member_sessions.items():
# Encrypt sender key using pairwise session
encrypted_key = session.encrypt(self.my_sender_key)
# Send to member via normal message path
yield member_id, encrypted_key
def encrypt_for_group(self, plaintext: bytes) -> bytes:
"""
Encrypt message using my sender key.
All group members who have my sender key can decrypt.
Much more efficient than pairwise encryption.
"""
# Use sender key with chain ratchet
message_key = self._derive_message_key()
ciphertext = self._aes_encrypt(message_key, plaintext)
# Ratchet forward
self._ratchet_sender_key()
return ciphertext
def handle_member_left(self, member_id: str):
"""
When someone leaves, all sender keys must be regenerated.
This ensures the departed member can't read future messages.
"""
# Generate new sender key
self.my_sender_key = self._generate_sender_key()
# Redistribute to remaining members
# (handled by caller)
What WhatsApp CAN See (Metadata):
ENCRYPTION SCOPE
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ ENCRYPTED (WhatsApp cannot read): │
│ ───────────────────────────────── │
│ • Message content │
│ • Media files (photos, videos, documents) │
│ • Voice notes │
│ • Voice and video call content │
│ • Status updates │
│ │
│ NOT ENCRYPTED (WhatsApp can see): │
│ ───────────────────────────────── │
│ • Who messaged whom (sender/recipient) │
│ • When messages were sent │
│ • Message size │
│ • Device information │
│ • Phone numbers │
│ • Profile photos │
│ • Group membership │
│ • Online/offline status │
│ │
│ This metadata is necessary for routing but reveals communication │
│ patterns even if content is secret. │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Deep Dive 3: Offline Message Handling
Week 3 concepts: Message queuing, delivery guarantees.
You: "Most users aren't online all the time. Offline message storage is critical."
OFFLINE MESSAGE FLOW
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ Alice sends message to offline Bob │
│ │
│ Alice WhatsApp Bob │
│ │ │ │ │
│ │ ① Send (encrypted) │ │ (offline) │
│ │─────────────────────────▶│ │ │
│ │ │ │ │
│ │ ② ACK (single ✓) │ │ │
│ │◀─────────────────────────│ │ │
│ │ "Message received │ │ │
│ │ by server" │ │ │
│ │ │ │ │
│ │ ③ Store in │ │
│ │ Offline Queue │ │
│ │ ┌─────────┐ │ │
│ │ │ Bob's │ │ │
│ │ │ Queue │ │ │
│ │ │ ─────── │ │ │
│ │ │ Msg 1 │ │ │
│ │ │ Msg 2 │ │ │
│ │ │ ... │ │ │
│ │ └─────────┘ │ │
│ │ │ │ │
│ │ │ │ │
│ │ ... TIME PASSES (Bob offline) ... │ │
│ │ │ │ │
│ │ │ │ │
│ │ │ ④ Bob comes online │ │
│ │ │◀─────────────────────────│ │
│ │ │ TCP connection │ │
│ │ │ │ │
│ │ │ ⑤ Drain queue │ │
│ │ │─────────────────────────▶│ │
│ │ │ Send all pending msgs │ │
│ │ │ │ │
│ │ ⑥ Delivered (✓✓) │ ⑦ Delivery ACK │ │
│ │◀─────────────────────────│◀─────────────────────────│ │
│ │ │ │ │
│ │ ⑧ Delete from │ │
│ │ queue (delivered) │ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ │
└──────────────────────────────────────────────────────────────────────────┘
# offline/message_store.py
"""
Offline Message Storage
WhatsApp's approach:
- Messages stored only until delivered
- After delivery confirmation, deleted from server
- This minimizes server storage and privacy exposure
"""
from dataclasses import dataclass
from typing import List, Optional
from datetime import datetime, timedelta
@dataclass
class QueuedMessage:
"""A message waiting for delivery."""
message_id: str
sender_id: str
recipient_id: str
encrypted_content: bytes
timestamp: datetime
expiry: datetime # Messages expire after 30 days
# Delivery tracking
delivery_attempts: int = 0
last_attempt: Optional[datetime] = None
class OfflineMessageStore:
"""
Stores messages for offline users.
Design principles:
- In-memory for speed (Mnesia ETS tables)
- Replicated for durability
- Partitioned by recipient for scalability
- Messages deleted after delivery
"""
def __init__(self, db, max_queue_size: int = 10000):
self.db = db # Mnesia in actual WhatsApp
self.max_queue_size = max_queue_size
# TTL for stored messages
self.message_ttl = timedelta(days=30)
async def queue_message(
self,
sender_id: str,
recipient_id: str,
encrypted_content: bytes
) -> QueuedMessage:
"""
Store message for offline recipient.
"""
message = QueuedMessage(
message_id=generate_id(),
sender_id=sender_id,
recipient_id=recipient_id,
encrypted_content=encrypted_content,
timestamp=datetime.utcnow(),
expiry=datetime.utcnow() + self.message_ttl
)
# Check queue size limits
queue_size = await self.db.count_for_recipient(recipient_id)
if queue_size >= self.max_queue_size:
# Drop oldest message (or reject new one)
await self.db.delete_oldest(recipient_id)
await self.db.save(message)
return message
async def get_pending_messages(
self,
recipient_id: str
) -> List[QueuedMessage]:
"""
Get all pending messages for a user who just came online.
Called when user establishes connection.
"""
messages = await self.db.get_by_recipient(
recipient_id,
order_by="timestamp",
limit=1000 # Batch size
)
# Filter expired messages
now = datetime.utcnow()
valid_messages = [m for m in messages if m.expiry > now]
return valid_messages
async def mark_delivered(
self,
message_ids: List[str]
):
"""
Delete messages that have been delivered.
This is called after receiving delivery ACK from recipient.
"""
for message_id in message_ids:
await self.db.delete(message_id)
async def cleanup_expired(self):
"""
Background job: Delete expired messages.
Runs periodically to clean up messages that
were never delivered (user never came back online).
"""
now = datetime.utcnow()
expired = await self.db.get_expired(now)
for message in expired:
await self.db.delete(message.message_id)
# Optionally notify sender that message expired
# (WhatsApp doesn't do this)
class WriteBackCache:
"""
WhatsApp uses write-back caching for offline messages.
- Messages stored in memory first (fast)
- Asynchronously persisted to disk
- Handles disk I/O slowdowns gracefully
"""
def __init__(self, memory_store, disk_store):
self.memory = memory_store # ETS table
self.disk = disk_store # Mnesia disk table
self.pending_writes = []
async def store(self, message: QueuedMessage):
"""
Store in memory immediately, persist async.
"""
# Immediate: memory storage
await self.memory.put(message.message_id, message)
# Async: queue for disk persistence
self.pending_writes.append(message)
# Trigger background write (non-blocking)
asyncio.create_task(self._flush_writes())
async def _flush_writes(self):
"""
Background: persist to disk.
Batched for efficiency.
"""
if len(self.pending_writes) >= 100:
batch = self.pending_writes[:100]
self.pending_writes = self.pending_writes[100:]
await self.disk.batch_write(batch)
Deep Dive 4: Presence and Typing Indicators
Week 4 concepts: Caching, real-time updates.
You: "Presence is the 'chattiest' part of the system — far more traffic than messages."
PRESENCE TRAFFIC ANALYSIS
For every message sent, there are ~10-100 presence events:
├── User opens app (online)
├── User starts typing (typing indicator)
├── User stops typing
├── User reads messages (last seen update)
├── User closes app (offline)
└── Repeated for every chat participant watching
With 500M daily active users:
├── Messages: ~100 billion/day
├── Presence events: ~1 trillion/day
└── This is 10x the message traffic!
Presence CANNOT go through the same path as messages.
It needs aggressive optimization.
# presence/manager.py
"""
Presence Management
Optimizations WhatsApp uses:
1. Batching: Aggregate multiple presence updates
2. Rate limiting: Don't send typing every keystroke
3. TTL: Presence expires automatically
4. Selective delivery: Only send to active chats
"""
from dataclasses import dataclass
from typing import Dict, Set, Optional
from datetime import datetime, timedelta
from enum import Enum
class PresenceState(Enum):
ONLINE = "online"
OFFLINE = "offline"
TYPING = "typing"
@dataclass
class UserPresence:
"""Current presence state for a user."""
user_id: str
state: PresenceState
last_seen: datetime
typing_in_chat: Optional[str] = None
class PresenceManager:
"""
Manages presence state and notifications.
Key optimizations:
1. In-memory only (no persistence needed)
2. TTL-based expiry (online expires in 60s)
3. Rate limiting (typing indicator throttled)
4. Subscription-based (only notify interested parties)
"""
def __init__(self, redis_cluster, notification_service):
self.redis = redis_cluster
self.notifier = notification_service
# TTLs
self.online_ttl = timedelta(seconds=60)
self.typing_ttl = timedelta(seconds=5)
async def set_online(self, user_id: str):
"""
Mark user as online.
Uses Redis with TTL — automatically expires to offline.
"""
presence = UserPresence(
user_id=user_id,
state=PresenceState.ONLINE,
last_seen=datetime.utcnow()
)
# Store with TTL (auto-expires to offline)
await self.redis.set(
f"presence:{user_id}",
presence.to_json(),
ex=int(self.online_ttl.total_seconds())
)
# Notify subscribers
await self._notify_subscribers(user_id, presence)
async def set_typing(self, user_id: str, chat_id: str):
"""
Mark user as typing in a specific chat.
Rate limited: max 1 update per second per chat.
"""
# Rate limit check
rate_key = f"typing_rate:{user_id}:{chat_id}"
if await self.redis.exists(rate_key):
return # Already sent recently, skip
# Set rate limit marker (1 second)
await self.redis.set(rate_key, "1", ex=1)
# Update typing state
await self.redis.set(
f"typing:{user_id}:{chat_id}",
"1",
ex=int(self.typing_ttl.total_seconds())
)
# Notify only the other participant(s) of this chat
await self._notify_chat_participants(
chat_id,
user_id,
PresenceState.TYPING
)
async def get_presence(self, user_id: str) -> Optional[UserPresence]:
"""
Get current presence for a user.
Used when opening a chat to show current status.
"""
data = await self.redis.get(f"presence:{user_id}")
if data:
return UserPresence.from_json(data)
else:
# No presence record = offline
# Get last seen from persistent storage
last_seen = await self._get_last_seen(user_id)
return UserPresence(
user_id=user_id,
state=PresenceState.OFFLINE,
last_seen=last_seen
)
async def _notify_subscribers(
self,
user_id: str,
presence: UserPresence
):
"""
Notify users who care about this user's presence.
Optimization: Only notify users who have this user's
chat currently open (not all contacts).
"""
# Get list of users with this chat open
subscribers = await self.redis.smembers(
f"presence_subscribers:{user_id}"
)
for subscriber_id in subscribers:
await self.notifier.send_presence_update(
subscriber_id,
presence
)
async def subscribe_to_presence(
self,
subscriber_id: str,
target_user_id: str
):
"""
Subscribe to presence updates for a user.
Called when user opens a chat.
"""
await self.redis.sadd(
f"presence_subscribers:{target_user_id}",
subscriber_id
)
# Auto-expire subscription after 10 minutes
await self.redis.expire(
f"presence_subscribers:{target_user_id}",
600
)
async def unsubscribe_from_presence(
self,
subscriber_id: str,
target_user_id: str
):
"""
Unsubscribe from presence updates.
Called when user closes a chat.
"""
await self.redis.srem(
f"presence_subscribers:{target_user_id}",
subscriber_id
)
Phase 5: Scaling and Edge Cases
Interviewer: "WhatsApp grew from 0 to 500 million users in 5 years. How did they scale without rewriting everything?"
You: "Erlang's design made horizontal scaling almost automatic. But they still faced challenges..."
WhatsApp's Scaling Journey
WHATSAPP SCALING TIMELINE
2009: Launch
├── Few servers
├── Basic XMPP protocol
└── Thousands of users
2011: 1 million users
├── Custom Erlang backend
├── Modified ejabberd
└── ~10 servers
2013: 200 million users
├── 2 million connections per server achieved
├── Custom BEAM patches
├── ~100 servers
2014 (acquisition): 500 million users
├── 50 billion messages/day
├── 550 servers
├── 32 backend engineers
2020: 2 billion users
├── 100+ billion messages/day
├── Thousands of servers
├── Still running on Erlang
KEY INSIGHT:
The architecture scaled 2000x (1M → 2B users) with
the same fundamental design. That's the power of
choosing the right tool for the job.
Edge Cases
EDGE CASE 1: Celebrity with 10M Followers Posts Status
Problem: Status update needs to reach 10 million people
Impact: Massive fan-out, could overwhelm system
Solution:
├── Status updates are pull-based, not push
├── User's device polls for updates when opening status tab
├── Updates cached at edge servers
├── Same update served to millions from cache
└── No per-follower push notification
EDGE CASE 2: Group with 1000 Members
Problem: Each message needs 1000 deliveries
Impact: 1 message = 1000x amplification
Solution:
├── Sender Key protocol (not pairwise encryption)
├── Message encrypted once, decrypted by all members
├── Group metadata (membership) changes trigger rekey
├── Stagger delivery to avoid thundering herd
└── Members who leave can't read future messages
EDGE CASE 3: Network Partition
Problem: User connected to Server A, recipient on Server B
Network between A and B fails
Solution:
├── Messages queued at Server A
├── Periodic retry to deliver to Server B
├── If partition heals, messages delivered
├── If partition persists, messages expire after 30 days
└── User notified of delivery failures after timeout
EDGE CASE 4: Device Goes Underwater (Literally)
Problem: User's phone destroyed, never comes back online
Impact: Their offline queue grows forever
Solution:
├── Messages expire after 30 days
├── Expired messages deleted from server
├── Sender never gets "delivered" status
├── No infinite queue growth
└── New device = new session, old messages lost
Phase 6: Why 50 Engineers?
You: "The small team is the most impressive part. Here's why it worked..."
WHY 50 ENGINEERS COULD BUILD WHATSAPP
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ 1. TECHNOLOGY CHOICES REDUCED COMPLEXITY │
│ ─────────────────────────────────────── │
│ │
│ Erlang's actor model: │
│ ├── No mutex/lock management (huge complexity source) │
│ ├── Built-in distribution (no Kafka/RabbitMQ to manage) │
│ ├── Hot code reloading (no complex deployment pipelines) │
│ └── Supervision trees (automatic failure recovery) │
│ │
│ FreeBSD: │
│ ├── Simpler than Linux (one distribution) │
│ ├── Better network stack for their use case │
│ └── Team had deep expertise (ex-Yahoo) │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 2. FOCUS REDUCED SCOPE │
│ ────────────────────── │
│ │
│ What WhatsApp DIDN'T build: │
│ ├── Advertising system (no ads = no ad tech team) │
│ ├── Social feed (no feed ranking team) │
│ ├── Recommendation engine (no ML team) │
│ ├── Third-party integrations (no platform team) │
│ ├── Multiple products (no product sprawl) │
│ └── Enterprise features (initially) │
│ │
│ Every feature NOT built = engineers NOT needed │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 3. COMMUNICATION OVERHEAD IS QUADRATIC │
│ ───────────────────────────────────── │
│ │
│ Team of 50: 50 × 49 / 2 = 1,225 communication paths │
│ Team of 500: 500 × 499 / 2 = 124,750 paths │
│ │
│ Larger teams spend more time coordinating than coding. │
│ Small teams move FASTER for well-scoped problems. │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 4. DEEP EXPERTISE > BROAD KNOWLEDGE │
│ ─────────────────────────────────── │
│ │
│ Every engineer knew: │
│ ├── The entire codebase │
│ ├── Erlang deeply (not just superficially) │
│ ├── FreeBSD kernel internals │
│ └── The complete message flow │
│ │
│ No "that's not my service" mentality. │
│ Anyone could debug any issue. │
│ │
│ ───────────────────────────────────────────────────────────────────── │
│ │
│ 5. CODE REVIEW BY FOUNDERS │
│ ────────────────────────── │
│ │
│ In early days, Jan Koum reviewed every line of backend code. │
│ This ensured: │
│ ├── Consistent style │
│ ├── No unnecessary complexity │
│ ├── Deep understanding by leadership │
│ └── Quality over velocity │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Interview Conclusion
Interviewer: "Excellent coverage. Quick questions:"
Interviewer: "If you were starting today, would you still use Erlang?"
You: "It depends. Erlang is still excellent for this use case, but:
Advantages of Erlang:
- Perfect for massive concurrency
- Battle-tested at WhatsApp scale
- 'Let it crash' philosophy reduces code complexity
Disadvantages:
- Smaller talent pool (hard to hire)
- Less library ecosystem than Go/Rust
- Learning curve for new engineers
Modern alternatives:
- Go: Good concurrency, larger talent pool
- Rust: Performance, growing ecosystem
- Elixir: Erlang VM with nicer syntax
For a startup, I'd probably choose Go for the talent pool. For WhatsApp's specific requirements, Erlang is still hard to beat."
Interviewer: "What's the hardest part of building a messaging system?"
You: "Three things:
-
Exactly-once delivery semantics — Messages should arrive once, in order, or clearly fail. Network unreliability makes this hard.
-
End-to-end encryption at scale — Key management, device verification, group re-keying. Most teams underestimate this complexity.
-
Presence at scale — The 'chattiest' part of the system. Easy to ignore, hard to scale. Most outages are presence-related."
Summary: Concepts Applied from 10-Week Course
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ CONCEPTS FROM 10-WEEK COURSE IN WHATSAPP DESIGN │
│ │
│ WEEK 1: DATA AT SCALE │
│ ├── Partitioning: Users partitioned across "islands" │
│ ├── Replication: Each island has a replica │
│ └── In-memory storage: ETS/Mnesia for speed │
│ │
│ WEEK 2: FAILURE-FIRST DESIGN │
│ ├── Let it crash: Erlang supervision trees │
│ ├── Process isolation: One crash doesn't affect others │
│ ├── Circuit breakers: Island isolation │
│ └── Retry: Message redelivery for transient failures │
│ │
│ WEEK 3: MESSAGING & ASYNC │
│ ├── Offline queues: Messages stored until delivered │
│ ├── At-least-once delivery: With deduplication at client │
│ └── Backpressure: Queue size limits │
│ │
│ WEEK 4: CACHING │
│ ├── Presence caching: In-memory with TTL │
│ ├── Write-back cache: For offline message persistence │
│ └── Edge caching: Media and status updates │
│ │
│ WEEK 5: CONSISTENCY │
│ ├── Eventual consistency: Presence can be stale │
│ ├── At-least-once: Messages, with client dedup │
│ └── Message ordering: Per-conversation, not global │
│ │
│ WEEK 9: SECURITY │
│ ├── End-to-end encryption: Signal Protocol │
│ ├── Forward secrecy: Double Ratchet algorithm │
│ ├── Key management: X3DH key agreement │
│ └── Zero-knowledge: Server can't read messages │
│ │
│ WEEK 10: OPERATIONS │
│ ├── Hot code reloading: Deploy without restart │
│ ├── Observability: Per-process stats in BEAM │
│ └── Small team: Simplicity enables velocity │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Why WhatsApp Matters
╔═════════════════════════════════════════════════════════════════════════╗
║ ║
║ WHY WHATSAPP IS AN ENGINEERING MARVEL ║
║ ║
║ EFFICIENCY ║
║ ────────── ║
║ • 2 million connections per server (20x industry norm) ║
║ • 50 billion messages/day with 32 engineers ║
║ • $19 billion valuation with 55 employees ║
║ ║
║ RELIABILITY ║
║ ─────────── ║
║ • 99.99%+ uptime ║
║ • Messages reliably delivered worldwide ║
║ • Works on 2G networks in developing countries ║
║ ║
║ PRIVACY ║
║ ─────── ║
║ • End-to-end encryption by default ║
║ • Even WhatsApp can't read your messages ║
║ • Signal Protocol: Gold standard for secure messaging ║
║ ║
║ SIMPLICITY ║
║ ────────── ║
║ • No ads, no social feed, no distractions ║
║ • One feature done exceptionally well ║
║ • Focus enabled small team to move fast ║
║ ║
║ ════════════════════════════════════════════════════════════════════ ║
║ ║
║ "The best systems are simple enough that a small team can ║
║ understand them completely, yet powerful enough to serve ║
║ billions of users." ║
║ ║
║ WhatsApp proved that architecture beats headcount. ║
║ ║
╚═════════════════════════════════════════════════════════════════════════╝
Self-Assessment Checklist
After studying this case study, you should be able to:
Architecture:
- Explain why Erlang is well-suited for messaging
- Design a connection handling system for millions of users
- Implement offline message queuing and delivery
Distributed Systems:
- Apply the "let it crash" philosophy
- Design supervision trees for failure recovery
- Handle network partitions gracefully
Security:
- Explain Signal Protocol at a high level
- Understand forward secrecy and its importance
- Design key exchange for end-to-end encryption
Performance:
- Optimize presence/typing for high frequency updates
- Use TTL-based expiry for ephemeral data
- Apply batching and rate limiting
Team Efficiency:
- Understand how technology choices affect team size
- Value simplicity over cleverness
- Focus on doing one thing well
Sources
Architecture and Scale:
- ByteByteGo - How WhatsApp Handles 40 Billion Messages: https://blog.bytebytego.com/p/how-whatsapp-handles-40-billion-messages
- GetStream - How WhatsApp Works Architecture: https://getstream.io/blog/whatsapp-works/
- High Scalability - WhatsApp Architecture Facebook Bought: https://highscalability.com/the-whatsapp-architecture-facebook-bought-for-19-billion/
- System Design One - 8 Reasons WhatsApp Scaled with 32 Engineers: https://newsletter.systemdesign.one/p/whatsapp-engineering
- Talent500 - How WhatsApp Powers 40 Billion Messages: https://talent500.com/blog/whatsapp-scalable-messaging-architecture/
- CometChat - WhatsApp Architecture and System Design: https://www.cometchat.com/blog/whatsapps-architecture-and-system-design
Erlang and FreeBSD:
- FreeBSD Foundation - WhatsApp Testimonial: https://freebsdfoundation.org/testimonial/whatsapp/
- FreeBSD Forums - WhatsApp on FreeBSD Discussion: https://forums.freebsd.org/threads/whatsapp-on-freebsd.87908/
- Erlang Factory - Rick Reed Speaker Profile: http://www.erlang-factory.com/conference/SFBay2012/speakers/RickReed
- Medium - WhatsApp Engineering Inside: https://medium.com/codingurukul/whatsapp-engineering-inside-2-bdd1ec354748
- Medium - WhatsApp's Billion-User Database with FreeBSD and Erlang: https://medium.com/@yashbatra11111/whatsapps-billion-user-database-how-freebsd-and-erlang-handled-the-impossible-5e699f7f078d
- SlideShare - WhatsApp's Architecture: https://www.slideshare.net/slideshow/whatsapps-architecture/57522146
End-to-End Encryption:
- Signal.org - WhatsApp Integration Complete: https://signal.org/blog/whatsapp-complete/
- Wikipedia - Signal Protocol: https://en.wikipedia.org/wiki/Signal_Protocol
- WhatsApp FAQ - About End-to-End Encryption: https://faq.whatsapp.com/820124435853543
- Gupshup - WhatsApp E2E Encryption Guide: https://www.gupshup.ai/resources/blog/whatsapp-end-to-end-encryption/
- Requestly Blog - WhatsApp Chat Security with E2EE: https://requestly.com/blog/how-whatsapp-ensures-chat-security-with-end-to-end-encryption/
- SDERay - Signal Protocol Engineering Deep Dive: https://sderay.com/how-end-to-end-encryption-works-the-engineering-behind-signal-whatsapp-security/
- Medium - WhatsApp E2E Encryption How It Works: https://medium.com/@panghalamit/whatsapp-s-end-to-end-encryption-how-does-it-work-80020977caa0
- MIT Course Paper - WhatsApp Security Analysis: https://courses.csail.mit.edu/6.857/2016/files/36.pdf
System Design Resources:
- GeeksforGeeks - How WhatsApp Handles 50 Billion Messages: https://www.geeksforgeeks.org/system-design/how-whatsapp-handles-50-billion-messages-a-day/
- Medium - WhatsApp System Design Complete Architecture: https://medium.com/@yadavsatale/whatsapp-system-design-a-complete-architecture-deep-dive-8949f8d4eb2b
- TRTC - Definitive Guide to WhatsApp System Design: https://trtc.io/blog/details/whatsapp-system
- Medium - System Design of WhatsApp for Android: https://medium.com/@YodgorbekKomilo/the-system-design-of-whatsapp-for-android-behind-the-scenes-of-a-global-messaging-giant-c80175b18016
- Medium - Unpacking WhatsApp's System Design: https://medium.com/@lovejot.singh/unpacking-whatsapps-system-design-the-power-behind-your-messages-316280b38f78
Further Reading
Official Documentation:
- WhatsApp Security Whitepaper: https://www.whatsapp.com/security/WhatsApp-Security-Whitepaper.pdf
- Signal Protocol Specifications: https://signal.org/docs/
- WhatsApp Engineering Blog: https://engineering.fb.com/ (Search for WhatsApp posts)
Conference Talks (Highly Recommended):
- Rick Reed - "Scaling to Millions of Simultaneous Connections" (Erlang Factory 2012)
- YouTube: Search "Rick Reed WhatsApp Erlang"
- Rick Reed - "That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp" (Erlang Factory 2014)
- Details on achieving 2M+ connections per server
- Anton Lavrik - "A Reflection on Building the WhatsApp Server" (Code BEAM 2018)
- Post-Facebook acquisition insights
- Eugene Fooksman - "WhatsApp System Design" (Various conferences)
Engineering Blogs:
- High Scalability Blog: https://highscalability.com/ (Multiple WhatsApp articles)
- ByteByteGo Newsletter: https://blog.bytebytego.com/ (System design breakdowns)
- System Design One: https://newsletter.systemdesign.one/ (Engineering case studies)
- The Pragmatic Engineer: https://newsletter.pragmaticengineer.com/ (Tech industry insights)
Erlang/BEAM Resources:
- Learn You Some Erlang: https://learnyousomeerlang.com/ (Free online book)
- Erlang.org Documentation: https://www.erlang.org/doc/
- BEAM Book: https://blog.stenmans.org/theBeamBook/ (BEAM VM internals)
- Elixir Forum: https://elixirforum.com/ (Modern Erlang VM discussions)
Cryptography and Security:
- Signal Protocol Documentation: https://signal.org/docs/
- Double Ratchet Algorithm: https://signal.org/docs/specifications/doubleratchet/
- X3DH Key Agreement Protocol: https://signal.org/docs/specifications/x3dh/
- Crypto101: https://www.crypto101.io/ (Free cryptography book)
Books:
- "Designing Data-Intensive Applications" by Martin Kleppmann
- Chapters on messaging and distributed systems
- "Programming Erlang" by Joe Armstrong (Erlang creator)
- Understanding the language that powers WhatsApp
- "Designing Elixir Systems with OTP" by James Edward Gray II
- Modern take on BEAM VM patterns
- "System Design Interview Vol 1 & 2" by Alex Xu
- Messaging system design patterns
Related Systems to Study:
- Telegram: Different architecture (MTProto protocol)
- Signal: Open-source reference implementation
- Discord: Modern take on real-time messaging
- Slack: Enterprise messaging architecture
Research Papers:
- "The Signal Protocol" - Formal security analysis papers
- "Asynchronous Ratcheting Trees" - Group messaging improvements
- "SoK: Secure Messaging" - Survey of secure messaging systems
Podcasts and Videos:
- Software Engineering Daily: Episodes on messaging systems
- InfoQ: Conference talks on distributed systems
- Strange Loop Conference: Talks on Erlang/BEAM
End of Bonus Problem 2: WhatsApp Messaging
"50 engineers. 2 billion users. 100 billion messages. The right architecture makes the impossible possible."