Zoom Video Meetings at Scale design sheet

At a glance#

Number	Value	What it drives
Daily meeting participants	10M (Dec 2019) → 300M (Apr 2020)	a count of joins, not people: one person in five meetings counts five times
Meeting minutes	3.5T annualized (Oct 2020) · ~9.6B a day	treated as participant-minutes; ~32 min per join
Concurrent participants	~6.7M average · ~20M at the peak hour	a 3× peak; ~3.3M meetings of 6
Media at peak	~15 Tbps in · ~30 Tbps out · ~3B packets / s	per-packet work must be a lookup and a send
Forward vs mix	~4,500 servers vs ~360,000	route, don't mix: mixing decodes every camera and encodes per receiver
Joins	~60,000 / s at the top of the hour · design for 100,000	the herd hits the control plane, not the media plane
Delay	< 150 ms one way in region (ITU-T G.114)	nearest zone per participant; one clean long hop between regions
Data centres	17 in 2020, 2 in India (Mumbai, Hyderabad)	meeting zones, 50% spare capacity
End-to-end encryption	up to 200 participants · no cloud recording	servers become relays without keys

Architecture#

DiagramA control plane and a forwarding media plane

%% title: A control plane and a forwarding media plane
flowchart TD
  CL@{ icon: "sd:phone", label: "Desktop, mobile and web clients<br/>audio + 3 video layers" } ==> API@{ icon: "sd:api", label: "API gateway<br/>join, auth, signalling sockets" }
  API --> MS["Meeting service<br/>passcode · waiting room · roles"]:::service
  MS --> CTRL["Meeting controller<br/>one owner per meeting, versioned events"]:::core
  CTRL --> ST@{ icon: "tech:redis", label: "Redis<br/>live state · leases with epochs" }
  CTRL --> ZS["Zone selector<br/>RTT · capacity · residency"]:::service
  ZS --> ZC@{ icon: "sd:server", label: "Zone controller<br/>MMR health and load" }
  ZC --> MMR["MMR<br/>forwards packets, picks layers"]:::hot
  CL ==>|"UDP · TLS 443 fallback"| MMR
  MMR <==>|"one copy per layer"| BB@{ icon: "sd:globe", label: "Backbone<br/>MMR-to-MMR cascade" }
  MMR -.-> REC@{ icon: "sd:worker", label: "Recorder<br/>a silent participant" }
  REC --> OBJ@{ icon: "sd:object-storage", label: "Object storage<br/>~360 TB a day" }
  class CL client
  class API,BB edge
  class ST,OBJ store
  class ZC,REC service

Route, don't mix: the MMR reads headers (sender, layer, audio level) and copies packets; pixels are decoded only on clients.
Control and media never share a path: a mute or an admit is an ordered event from one controller; media is lossy UDP.
Place participants, not meetings: each joins its nearest zone; the meeting's MMRs cascade across regions.

DiagramOne sender, a layer per receiver

%% title: One sender, a layer per receiver
flowchart TD
  K@{ icon: "sd:laptop", label: "Sender<br/>180p 150 kbps · 360p 400 kbps · 720p 1.5 Mbps" }
  MMR["MMR<br/>budget per receiver: audio, share, speaker, thumbnails"]:::hot
  BWE["Bandwidth estimate per downlink<br/>arrival-time feedback · GCC-style rules"]:::core
  A@{ icon: "sd:phone", label: "Phone on 4G<br/>~600 kbps · 360p" }
  R@{ icon: "sd:laptop", label: "Laptop on fibre<br/>~20 Mbps · 720p" }
  T@{ icon: "sd:desktop", label: "Gallery tiles<br/>180p" }
  K ==> MMR
  BWE -.-> MMR
  MMR ==> A
  MMR ==> R
  MMR ==> T
  A -.->|"feedback"| BWE
  class K,A,R,T client

Key flows#

DiagramKavya joins, then speaks

%% title: Kavya joins, then speaks
sequenceDiagram
  autonumber
  participant K as Kavya's app
  participant A as API gateway
  participant C as Meeting controller
  participant Z as Zone selector
  participant M as MMR, Mumbai zone
  participant R as MMR, San Jose zone
  K->>A: Join with meeting ID and passcode
  A->>C: Admit Kavya (host)
  C->>Z: Nearest zone with room
  Z-->>K: Mumbai MMR, media token, 60 s
  K->>M: UDP hello with token
  K-)M: Opus audio, 3 video layers
  M-)R: One copy of each layer San Jose needs
  Note over K,M: about 1.5 s from click to first audio

DiagramArjun's cab enters a dead spot

%% title: Arjun's cab enters a dead spot
sequenceDiagram
  autonumber
  participant A as Arjun's phone
  participant M as MMR
  A-)M: Arrival times, several reports a second
  M->>M: Delay rising, estimate falls to 600 kbps
  M-)A: Kavya at 360p, thumbnails paused, audio kept
  A-)M: NACK for missing video packets
  M-)A: Resent from a short buffer
  Note over A: audio rebuilt from Opus in-band FEC
  Note over M: Rahul and the others stay on 720p

DiagramMeenal behind a firewall that allows only HTTPS

%% title: Meenal behind a firewall that allows only HTTPS
sequenceDiagram
  autonumber
  participant C as Meenal's client
  participant P as Corporate proxy
  participant M as MMR
  C->>M: UDP to the media ports
  C->>P: TLS on 443, raced in parallel
  P->>M: Tunnelled media session
  M-->>C: TLS works first, UDP never answers
  Note over C,M: same meeting, more delay under loss, lower layers sooner

Data model#

DiagramCore entities

%% title: Core entities
classDiagram
  class Meeting {
    +meetingId : string
    +hostId : user
    +passcode : hash
    +waitingRoom : bool
    +locked : bool
    +residency : region list
  }
  class Participant {
    +participantId : string
    +sessionId : survives reconnects
    +role : host / co-host / attendee
    +muted : bool
    +zone : meeting zone
  }
  class MediaToken {
    +meeting : meetingId
    +participant : participantId
    +role : string
    +expires : about 60 s
  }
  class Subscription {
    +receiver : participantId
    +sender : participantId
    +tilePx : int
    +layer : 180p / 360p / 720p / paused
  }
  class ControllerLease {
    +meetingId : string
    +epoch : int, fencing
    +version : last event
  }
  class Recording {
    +meetingId : string
    +chunks : object keys
    +retention : days
  }
  Meeting "1" --> "many" Participant : roster
  Participant "1" --> "1" MediaToken : admitted with
  Participant "1" --> "many" Subscription : receives
  Meeting "1" --> "1" ControllerLease : owned under
  Meeting "1" --> "0..many" Recording : produces

Meeting state is ~20 KB a meeting; ~3.3M live meetings is ~66 GB, in memory across controllers.
The MMR keeps only subscriptions and short retransmission buffers; losing one costs seconds, not the meeting.

Lifecycle#

DiagramA participant's life in a meeting

%% title: A participant's life in a meeting
stateDiagram-v2
  [*] --> JOINING
  JOINING --> WAITING: waiting room on
  JOINING --> IN_MEETING: admitted by policy
  WAITING --> IN_MEETING: host admits, token issued
  WAITING --> [*]: host denies
  IN_MEETING --> RECONNECTING: network change or MMR failure
  RECONNECTING --> IN_MEETING: same session within 60 s
  RECONNECTING --> [*]: timed out
  IN_MEETING --> [*]: leaves or removed, token revoked
  class IN_MEETING ok
  class WAITING queue

Decisions#

Decision	Chose	Over	Because
Media topology	SFU: forward packets, never decode	MCU mixing	~80× fewer servers, no added decode delay, a layout per receiver, E2EE possible
Small meetings	SFU too	Peer-to-peer mesh	10 people means 9 uploads from a phone: ~11 Mbps up
Adaptation	Simulcast or SVC layers, chosen per receiver	Re-encoding per receiver	the server can't decode; layers let it pick by dropping packets
Audio	Forward the 3 loudest by audio-level header	All streams to all	silence costs nothing; still no decoding
Transport	UDP with FEC, NACK, adaptive jitter buffer	TCP everywhere	a late packet is useless; TCP stalls behind a loss
Firewalls	Race UDP and TLS on 443	A slow sequential ladder	joins stay under a few seconds for blocked networks
Placement	Nearest zone per participant, cascaded MMRs	One MMR per meeting	each stream crosses an ocean once, on a clean backbone
Meeting state	One controller per meeting, lease + epoch fencing	State spread across MMRs	ordered host decisions; a stale owner can't admit anyone

When it breaks#

Failure	What users see	Contain it with
Top of the hour, ~60,000 joins / s	slow joins	pre-warm from yesterday's curve; one-round-trip join; jittered retries
An MMR dies with 400 people	a few seconds of freeze	reconnect with token and session to another MMR in the zone; state is in the controller
A keyframe storm in a 5,000-person webinar	frozen tiles at every speaker change	collapse requests at each MMR; pre-subscribe likely speakers at a low layer
4G loss in bursts	crackle, frozen video	FEC up as loss rises; NACK on short round trips; lower layers
UDP blocked	slow join, more delay	TLS on 443 raced in parallel; bigger jitter buffer
Uninvited guests	strangers sharing screens	passcodes and waiting room on by default; revoke tokens at the MMR
Controller failover	host controls pause a few seconds	new epoch rebuilds from snapshot and reconnects; media keeps flowing
A 40,000-person town hall	joins and keyframe requests pile onto one meeting	a tree of MMRs per region; no roster to attendees; degrade attendee video first

Say it in the interview#

"Separate control from media." Control is small, ordered and owned by one controller per meeting; media is lossy UDP through routers.
Route, don't mix: forwarding reads headers and copies packets; mixing would need ~360,000 servers instead of ~4,500 at peak.
Layers per receiver: senders send simulcast or SVC layers; the MMR fills each receiver's bandwidth budget from audio down.
Late is the same as lost: FEC and in-band FEC for audio, NACK when the round trip is short, an adaptive jitter buffer, TLS on 443 when UDP is blocked.
Place participants, cascade meetings: nearest zone for each person; one copy of each stream per region link.
Fence the controller: leases with epochs, versioned events, snapshots for clients that miss one.
Recording and E2EE don't mix: a recorder must see the media; E2EE exists so no server can (200 participants, no cloud recording).

Zoom Video Meetings at Scale, the whole design.