WhatsApp Messaging, the whole design.

%% title: WhatsApp on one page: phones → chat servers → recipient or offline queue
flowchart TD
  PH@{ icon: "sd:phone", label: "Android · iOS · Web clients<br/>encrypt on device · Signal Protocol" } -->|"persistent TCP + TLS<br/>custom binary protocol"| LB@{ icon: "sd:load-balancer", label: "GeoDNS + load balancers<br/>nearest data center" }
  PH -.->|"fetch public keys"| KS@{ icon: "sd:key", label: "Key server<br/>identity · signed + one-time pre-keys" }
  PH -.->|"upload encrypted, send a reference"| MS@{ icon: "sd:object-storage", label: "Media blob store<br/>30-day retention · dedup" }
  LB ==> CS["Chat servers · Erlang + FreeBSD<br/>1 process per connection · 1–2M conns each"]:::hot
  CS ==> UR@{ icon: "sd:database", label: "User registry · group metadata<br/>user → chat server · Mnesia + ETS" }
  CS -->|"online · typing · last seen"| PR@{ icon: "sd:cache", label: "Presence store<br/>in memory only · 60 s TTL" }
  UR ==>|"recipient online"| CS2["Recipient's chat server<br/>direct process-to-process send"]:::core
  UR -->|"recipient offline"| OQ@{ icon: "sd:queue", label: "Offline queue per recipient<br/>delete on ACK · 30-day expiry" }
  CS2 ==> BOB@{ icon: "sd:phone", label: "Bob's device<br/>delivered ACK → double tick" }
  OQ -.->|"drain on reconnect"| CS2
  class PH,BOB client
  class LB edge
  class KS,MS,UR store
  class PR cache
  class OQ queue

%% title: Inside a chat server: one supervised process per connection
flowchart TD
  VM["BEAM VM on one chat server<br/>2M processes ≈ 4 GB RAM"]:::hot --> SUP["Supervisor<br/>monitors children"]:::core
  VM -.-> ISL@{ icon: "sd:database", label: "Island · Mnesia node pair<br/>one slice of users · replicated in the pair" }
  SUP --> PA["User A process · ~2 KB<br/>socket · session · mailbox"]:::service
  SUP --> PB["User B process<br/>crashes"]:::bad
  SUP --> PC["User C process · ~2 KB"]:::service
  PB -->|"restart in milliseconds"| PB2["User B process, fresh<br/>B reconnects · A and C never notice"]:::ok
  class ISL store

%% title: Online send: Alice → Bob, one tick, two ticks, blue ticks
sequenceDiagram
  autonumber
  participant A as Alice's phone
  participant S as Chat server (Alice)
  participant R as User registry
  participant T as Chat server (Bob)
  participant B as Bob's phone
  A->>A: encrypt with Bob's session key
  A->>S: send blob · to bob · message ID
  S-->>A: ACK · SENT (single tick)
  S->>R: where is Bob connected?
  R-->>S: server 7
  S->>T: forward to Bob's process
  T->>B: deliver blob
  B-->>T: delivered ACK
  T-->>A: DELIVERED (double tick)
  B-->>A: read receipt via servers · READ (blue ticks)
  Note over S,T: server never decrypts · under 500 ms online

%% title: Bob was offline: store, drain, survive a dropped sync
sequenceDiagram
  autonumber
  participant A as Alice
  participant S as Chat server
  participant Q as Bob's offline queue
  participant B as Bob
  A->>S: 40 messages (Bob offline)
  S-->>A: single ticks
  S->>Q: store · message ID + per-chat sequence
  B->>S: reconnects
  S->>B: drain queue in batches
  alt connection drops mid-sync
    B--xS: lost before ACK
    Note over S,Q: nothing deleted without an ACK
    B->>S: reconnects again
    S->>B: resend unacked (at-least-once)
    Note over B: drop IDs already seen · order by sequence
  else sync completes
    B-->>S: delivery ACKs
  end
  S->>Q: delete acked messages
  S-->>A: double ticks
  Note over Q: never acked in 30 days → expired, no double tick

%% title: Group of 1,000: Sender Key, encrypt once
sequenceDiagram
  autonumber
  participant A as Alice (sender)
  participant S as Chat servers
  participant M as Each member
  A->>S: her sender key, encrypted pairwise per member (once)
  S->>M: deliver sender key
  A->>S: message encrypted once with sender key
  S->>M: deliver the same ciphertext to each member (staggered)
  Note over A,M: 1 encryption instead of 1,000 · server still delivers 1,000 copies
  alt a member leaves
    A->>S: new sender key to remaining members
    Note over M: the leaver can't read later messages
  end

%% title: Core entities
classDiagram
  class Account {
    +phone : string
    +devices : up to 4
    +privacy : hide last seen
  }
  class Device {
    +deviceId : string
    +identityKey : public key
    +signedPreKey : public key · rotates
    +oneTimePreKeys : ~100 · used once
  }
  class QueuedMessage {
    +messageId : string · dedup key
    +senderId : string
    +recipientDevice : string · partition key
    +blob : bytes · ~500 B encrypted
    +seq : int · per conversation
    +expiry : timestamp · 30 days
    +deliveryAttempts : int
  }
  class Group {
    +groupId : string
    +members : up to 1024
    +senderKeyVersion : int · rekey on leave
  }
  class Presence {
    +userId : string
    +state : ONLINE / TYPING / OFFLINE
    +lastSeen : timestamp
    +ttl : 60 s · memory only
  }
  class MediaObject {
    +ref : string · sent in message
    +blob : encrypted bytes
    +retention : 30 days
  }
  Account "1" --> "many" Device : owns
  Device "1" --> "many" QueuedMessage : offline queue
  Group "many" --> "many" Account : members
  QueuedMessage "many" --> "0..1" MediaObject : references
  Account "1" --> "1" Presence : shows

%% title: Message state machine (as the server sees it)
stateDiagram-v2
  [*] --> ENCRYPTED: composed on device
  ENCRYPTED --> SENT: server ACK · single tick
  SENT --> DELIVERED: recipient online, device ACK
  SENT --> QUEUED: recipient offline
  QUEUED --> QUEUED: dropped mid-sync, resend
  QUEUED --> DELIVERED: drained, device ACK
  QUEUED --> EXPIRED: 30 days, no ACK
  DELIVERED --> READ: read receipt · blue ticks
  DELIVERED --> [*]: deleted from server
  READ --> [*]
  EXPIRED --> [*]
  class READ ok
  class EXPIRED bad
  class QUEUED queue