Zoom Video Meetings at Scale.

How Do You Put Fourteen People on Three Continents Into One Room, Twenty Million Times at Once?

It is 9:58 on a Monday morning in Bengaluru. Kavya, a product manager, opens the link for her team's weekly review. Fourteen people will join in the next three minutes: Arjun on Jio 4G in a cab crawling down Baner Road in Pune, Meenal from a bank's office in Gurugram whose firewall blocks everything except web traffic, two engineers in Singapore, and Rahul in San Jose, where it is 9:28 on Sunday night.

At 10:00:00, Kavya isn't the only one. Across India, standups, classes and client calls all start on the hour, and the join button is pressed a few hundred thousand times in the same minute. Every one of those people expects to hear the others within a fraction of a second, see faces that don't freeze when a cab goes under a flyover, and never once think about which server is carrying their voice.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company building a video meeting product. The interviewer shares their screen: a gallery of fourteen faces, one of them pixelated, one frozen mid-blink, one a grey tile that says "Connecting…".

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Media Estimation#

Server Estimation#

Join and Control Estimation#

Key Metrics Summary#

System Architecture#

%% title: Video meetings: a control plane and a forwarding media plane
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    DESK@{ icon: "sd:desktop", label: "Desktop app" }
    MOB@{ icon: "sd:phone", label: "Android · iOS app" }
    WEB@{ icon: "tech:webrtc", label: "Web client<br/>WebRTC data channels" }
  end
  subgraph L2["Layer 2 · Control plane"]
    direction LR
    API@{ icon: "sd:api", label: "API gateway<br/>join, auth, signalling sockets" }
    MS["Meeting service<br/>passcode, waiting room, roles"]:::service
    CTRL["Meeting controller<br/>one owner per meeting"]:::core
    ZS["Zone selector<br/>geolocation + capacity"]:::service
    ST@{ icon: "tech:redis", label: "Redis<br/>live meeting state, leases" }
    DB@{ icon: "sd:database", label: "Meetings DB<br/>schedules, settings" }
  end
  subgraph L3["Layer 3 · Media plane, per zone"]
    direction LR
    ZC@{ icon: "sd:server", label: "Zone controller<br/>MMR health and load" }
    MMR["MMR<br/>forwards packets, picks layers"]:::hot
    CAS@{ icon: "sd:globe", label: "Backbone<br/>MMR-to-MMR cascade" }
  end
  subgraph L4["Layer 4 · After the meeting"]
    direction LR
    REC@{ icon: "sd:worker", label: "Recorder<br/>a silent participant" }
    OBJ@{ icon: "sd:object-storage", label: "Object storage<br/>recordings" }
  end
  DESK ==> API
  MOB ==> API
  WEB ==> API
  API --> MS --> CTRL
  MS --> DB
  CTRL --> ST
  CTRL --> ZS --> ZC
  ZC --> MMR
  DESK ==>|"UDP media"| MMR
  MOB ==>|"UDP media"| MMR
  WEB ==>|"data channel"| MMR
  MMR <==> CAS
  MMR -.-> REC --> OBJ
  class DESK,MOB,WEB client
  class API edge
  class ST,DB,OBJ store
  class ZC,REC service
  class CAS edge

KEY INSIGHTS:

• Route, don't mix. The MMR reads packet headers and decides which layer of which sender goes to which receiver; the pixels are only ever decoded on the clients
• Control and media never share a path. A mute or an admit is a small, ordered message through the meeting controller; media is a firehose that tolerates loss
• Place each participant, not each meeting. Everyone connects to their nearest zone, and the meeting spans zones through a cascade over a managed backbone

Data Flow: Kavya Joins, Then Speaks#

%% title: Kavya joins the review, then speaks
sequenceDiagram
  autonumber
  participant K as Kavya's app
  participant A as API gateway
  participant C as Meeting controller
  participant Z as Zone selector
  participant M as MMR, Mumbai zone
  participant R as Rahul's MMR, San Jose
  K->>A: Join 841 2290 7731, passcode
  A->>C: Admit Kavya (host)
  C->>Z: Nearest zone with room
  Z-->>K: Mumbai MMR, media token, 60 s
  K->>M: UDP hello with token
  M-->>K: Layers available from 13 senders
  Note over K,M: about 1.5 s from click to first audio
  K-)M: Opus audio, 3 video layers
  M-)R: One copy of each layer Rahul's side needs
  R-)R: Fan out to Rahul's receivers

MEANWHILE, IN PARALLEL:

• Every receiver reports what it got and when; each MMR estimates every downlink and picks layers per receiver several times a second
• The meeting controller pushes roster changes ("Arjun joined", "Meenal is muted") to every client over its signalling socket, in order

Deep Dive 1: Mesh, MCU or SFU, and Why Route Rather Than Mix (Week 10 Concepts: Capacity Planning; Week 1: Hot Keys and Skew)#

This builds on Week 10, Day 4 · Capacity Planning.

What is published: Zoom calls its media server a multimedia router and describes "multiple simultaneous streams" from which the app selects a layer. Zoom's web client, as Philipp Hancke found in 2018 and 2019, first carried media over WebSockets and later over WebRTC data channels, running Zoom's own codecs compiled to WebAssembly: the browser was another client of the same forwarding servers.

%% title: Mesh, mixing and forwarding for one meeting
flowchart TD
  subgraph MESH["Mesh"]
    direction LR
    A1@{ icon: "sd:phone", label: "Arjun<br/>uploads 13 copies" }
    O1["13 other clients"]:::bad
  end
  subgraph MCU["MCU · mixing"]
    direction LR
    A2@{ icon: "sd:phone", label: "Arjun<br/>uploads 1" }
    MIX["Mixer<br/>decode 10, encode 14 composites"]:::bad
  end
  subgraph SFU["SFU · forwarding"]
    direction LR
    A3@{ icon: "sd:phone", label: "Arjun<br/>uploads 1, three layers" }
    FWD["MMR<br/>copies packets, picks layers"]:::ok
  end
  A1 ==> O1
  A2 ==> MIX
  A3 ==> FWD
  class A1,A2,A3 client

• Audio is different. Forwarding 13 audio streams to every receiver is wasteful when only one or two people talk. The MMR reads each packet's audio level (an RTP header extension the sender fills in), forwards only the 3 loudest speakers, and the client mixes those. Still no decoding on the server.
• The active speaker is chosen the same way: a smoothed audio level per participant, with a little hysteresis, so a cough doesn't steal the big tile.
• Gallery view is a subscription. A receiver tells the MMR which tiles are on its screen and how big. A thumbnail off-screen costs nothing; a 49-tile gallery asks for 49 small layers, not 49 large ones.
• Big meetings are hot keys. A 1,000-person meeting can't fit on one MMR's network card if everyone watches everyone. Split it across several MMRs in the zone, linked like a cascade, so no single server holds the whole meeting.

# mmr/forward.py

"""
The heart of a forwarding media server (illustrative). It never decodes: it reads
the RTP header and a few header extensions (sender, layer, audio level), then
copies the packet to each receiver whose subscription wants that layer of that
sender. Per-packet work is a dictionary lookup and a send.
"""

from dataclasses import dataclass, field

TOP_SPEAKERS = 3


@dataclass
class Receiver:
    rid: str
    wants: dict = field(default_factory=dict)     # sender_id -> layer the selector chose
    send: callable = None


class MeetingRoute:
    def __init__(self):
        self.receivers: dict[str, Receiver] = {}
        self.loudest: list[str] = []              # updated every 100 ms from audio levels

    def on_video(self, pkt) -> None:
        for r in self.receivers.values():
            if r.rid != pkt.sender and r.wants.get(pkt.sender) == pkt.layer:
                r.send(pkt.rewrite_for(r.rid))      # SSRC and sequence rewritten per receiver

    def on_audio(self, pkt) -> None:
        if pkt.sender not in self.loudest:
            return                                  # silence and background voices aren't forwarded
        for r in self.receivers.values():
            if r.rid != pkt.sender:
                r.send(pkt.rewrite_for(r.rid))

    def update_loudest(self, levels: dict) -> None:
        """levels: sender -> smoothed audio level in dBov (closer to 0 is louder)."""
        self.loudest = sorted(levels, key=levels.get, reverse=True)[:TOP_SPEAKERS]

Deep Dive 2: Simulcast, SVC and Picking a Layer for Every Receiver (Week 3 Concepts: Backpressure and Flow Control)#

This builds on Week 3, Day 3 · Backpressure and Flow Control.

What is published: Zoom describes "multiple simultaneous streams" with the app selecting "the most appropriate layer". The standard pieces for doing this in RTP are written down: simulcast in SDP and RTP (RFC 8853), the RTP payload format for H.264 SVC (RFC 6190), and Google Congestion Control (draft-ietf-rmcat-gcc), whose loss-based rule keeps the estimate unchanged between 2% and 10% loss, raises it 5% below 2%, and cuts it by half the loss ratio above 10%, while a delay-based controller watches the variation in packet arrival times to catch a queue before it overflows.

%% title: One sender, three layers, a layer per receiver
flowchart TD
  K@{ icon: "sd:laptop", label: "Kavya's laptop<br/>180p · 360p · 720p" }
  MMR["MMR<br/>layer per receiver"]:::hot
  BWE["Bandwidth estimator<br/>per downlink, from arrival-time feedback"]:::core
  AR@{ icon: "sd:phone", label: "Arjun, 4G in a cab<br/>~600 kbps: 360p" }
  RA@{ icon: "sd:laptop", label: "Rahul, San Jose<br/>~20 Mbps: 720p" }
  TH@{ icon: "sd:desktop", label: "Gallery viewers<br/>small tile: 180p" }
  K ==> MMR
  BWE -.-> MMR
  MMR ==> AR
  MMR ==> RA
  MMR ==> TH
  AR -.->|"feedback"| BWE
  class K,AR,RA,TH client

• Simulcast or SVC? With these rates, simulcast costs the sender about a third more upload than the top layer alone and needs a keyframe to switch layers; SVC costs a little coding efficiency but lets the server drop layers on any packet boundary. Both work; the server logic is the same: a budget per receiver, filled from the most important stream down.
• Fill the budget by importance: audio first, always; then screen share; then the active speaker; then thumbnails, smallest layer first. Arjun at 600 kbps gets all audio, the speaker at 360p and thumbnails at 180p only if room remains.
• Probe before upgrading. Moving Arjun back up to 720p could overflow his link. Send padding or FEC packets at the higher rate for a second first; upgrade only if delay stays flat.
• Hysteresis: downgrade fast (within ~1 s of rising delay), upgrade slowly (after several seconds of headroom), so a receiver doesn't flicker between layers.

# mmr/layer_select.py

"""
Per-receiver layer selection (illustrative). Every ~200 ms, given the receiver's
estimated downlink and what is on its screen, choose a layer per sender: audio is
reserved first, then streams in order of importance, each at the best layer that
fits what's left and the tile it will be drawn in.
"""

LAYERS = [("q", 180, 150), ("h", 360, 400), ("f", 720, 1500)]     # (name, height px, kbps)
AUDIO_KBPS = 40


def select(estimate_kbps: float, tiles: list, speakers: int = 3) -> dict:
    """tiles: [(sender_id, tile_height_px, importance)], most important first."""
    budget = estimate_kbps * 0.9 - speakers * AUDIO_KBPS       # 10% headroom, audio reserved
    chosen = {}
    for sender, tile_px, _ in sorted(tiles, key=lambda t: -t[2]):
        fit = None
        for name, height, kbps in LAYERS:
            if kbps <= budget and height <= max(tile_px * 1.5, 180):
                fit = (name, kbps)                                  # best layer that fits so far
        if fit is None:
            chosen[sender] = None                                   # paused: show the avatar
            continue
        chosen[sender] = fit[0]
        budget -= fit[1]
    return chosen


# Arjun at 600 kbps, speaker tile 360 px, two thumbnails of 90 px:
# select(600, [("kavya", 360, 3), ("wei", 90, 1), ("rahul", 90, 1)]) -> {"kavya": "h", "wei": None, "rahul": None}

Deep Dive 3: UDP, Packet Loss, Jitter Buffers and the Way Through Firewalls (Week 2 Concepts: Timeouts and Retries)#

This builds on Week 2, Day 1 · Timeout Hell.

What is published: Zoom's network guidance lists UDP ports 8801–8810 for media and TCP ports 443, 8801 and 8802, and its architecture notes say UDP is used when available "with seamless fallback to TCP/TLS (including HTTPS/443)". The standards behind the recovery tools are RTCP feedback with NACK and picture-loss indication (RFC 4585), generic FEC for RTP (RFC 5109) and Opus, whose specification includes in-band FEC (RFC 6716).

%% title: Connecting through whatever the network allows
flowchart TD
  C@{ icon: "sd:laptop", label: "Meenal's client<br/>races the options" }
  U["UDP to the MMR<br/>media ports"]:::ok
  T["TCP to the MMR<br/>media ports"]:::service
  S["TLS on 443<br/>looks like HTTPS"]:::service
  P@{ icon: "sd:firewall", label: "Corporate proxy<br/>HTTPS only" }
  MMR["MMR"]:::hot
  C -->|"race: UDP"| U
  C -->|"fallback: TCP"| T
  C -->|"fallback: TLS 443"| S
  S ==> P ==> MMR
  U -.->|"preferred"| MMR
  class C client
  class P edge

• Race, don't queue the attempts. Start UDP and TLS together and keep whichever works first, upgrading to UDP later if it starts working. A sequential ladder with generous timeouts can add several seconds to a join.
• Keep the media session independent of the transport. When Arjun's phone hops from Wi-Fi to 4G, his IP address changes. The client reconnects with its media token and session ID, and the MMR swaps the address under the same participant: a second of silence, not a rejoin.
• Protect audio above everything. Audio is ~40 kbps; doubling it with FEC is cheap insurance. Video is where you save bandwidth.
• Measure loss after repair, not before. 8% loss in the network that becomes 0.3% after FEC and NACK is fine; that residual number is what users hear.

# client/jitter_buffer.py

"""
An adaptive jitter buffer for audio (illustrative). Packets are held until their
playout time; the target delay tracks a high percentile of recent delay variation,
so a smooth network gets a small buffer and a jittery one a larger buffer. Missing
frames are concealed (FEC copy if present, otherwise the decoder's guess).
"""

from collections import deque

FRAME_MS = 20


class JitterBuffer:
    def __init__(self, min_ms: int = 20, max_ms: int = 200):
        self.min_ms, self.max_ms = min_ms, max_ms
        self.target_ms = 40
        self.jitter = deque(maxlen=250)           # last ~5 s of samples
        self.packets = {}                         # seq -> payload

    def on_packet(self, seq: int, sent_ms: float, arrived_ms: float, payload) -> None:
        self.jitter.append(arrived_ms - sent_ms)  # relative one-way delay
        self.packets[seq] = payload
        if len(self.jitter) >= 50:
            base = min(self.jitter)
            spread = sorted(d - base for d in self.jitter)[int(len(self.jitter) * 0.95)]
            self.target_ms = max(self.min_ms, min(self.max_ms, spread + FRAME_MS))

    def play(self, seq: int):
        if seq in self.packets:
            return self.packets.pop(seq)
        nxt = self.packets.get(seq + 1)
        if nxt is not None and nxt.fec_copy is not None:
            return nxt.fec_copy                   # Opus in-band FEC rebuilds the lost frame
        return None                               # decoder conceals a 20 ms gap

Deep Dive 4: Meeting Placement and Cascading Across Regions (Week 1 Concepts: Partitioning; Week 9: Data Residency)#

This builds on Week 1, Day 1 · Partitioning Deep-Dive and Week 9, Day 3 · Data Residency.

What is published: Zoom's notes describe meeting zones duplicated in each data centre "with the exact same architecture", zone controllers that report MMR status to a global cloud controller, and clients steered by geolocation to the nearest available resources. In 2020 Zoom said it had 17 co-located data centres, nine of them in Equinix facilities as of November 2019, and two in India that had grown four times in capacity in four months.

%% title: One meeting across three regions
flowchart TD
  subgraph IN["India · Mumbai zone"]
    direction LR
    KV@{ icon: "sd:users", label: "Kavya and 10 others" }
    M1["MMR Mumbai<br/>meeting origin"]:::hot
  end
  subgraph SG["Singapore zone"]
    direction LR
    M2["MMR Singapore"]:::service
    SGU@{ icon: "sd:users", label: "2 engineers" }
  end
  subgraph US["San Jose zone"]
    direction LR
    M3["MMR San Jose"]:::service
    RH@{ icon: "sd:laptop", label: "Rahul" }
  end
  BB@{ icon: "sd:globe", label: "Backbone<br/>one copy per layer, per link" }
  KV ==> M1
  M1 <==> BB
  BB <==> M2
  BB <==> M3
  M2 ==> SGU
  M3 ==> RH
  class KV,SGU,RH client
  class BB edge

• The zone selector scores zones, not just distance: measured round trip from the client's region, free capacity in the zone (keep the 50% headroom), the account's residency rule, and whether the meeting already has an MMR there. Joining a zone the meeting is already in saves a cascade link.
• Cascades are trees, not meshes. With five regions, linking every MMR to every other multiplies the links; a tree rooted where most speakers are keeps each stream crossing each link once. Re-root if the speakers move.
• Webinars are the same tree, wider. Panelists sit on an interactive tier; tens of thousands of view-only attendees hang off leaf MMRs that receive only the speaker, the screen share and audio. Zoom said in 2024 its webinars reach up to 1 million attendees with 1,000 video panelists, without the multi-second delay of a streaming service.
• A full MMR splits the meeting, it doesn't refuse it: a second MMR in the same zone joins the cascade and new participants land there.

# placement/zone_select.py

"""
Choosing a meeting zone for one participant (illustrative). Prefer zones the
meeting already uses, then low round trip, while respecting residency and keeping
each zone's spare capacity.
"""

from dataclasses import dataclass

HEADROOM = 0.5          # keep half of each zone's capacity free
SAME_MEETING_MS = 40    # a zone already in the meeting is worth 40 ms of extra distance


@dataclass
class Zone:
    name: str
    country: str
    capacity_gbps: float
    used_gbps: float


def pick_zone(rtt_ms: dict, zones: list, meeting_zones: set, allowed_countries=None) -> str:
    best, best_score = None, float("inf")
    for z in zones:
        if allowed_countries and z.country not in allowed_countries:
            continue                                          # data residency
        if z.used_gbps > z.capacity_gbps * (1 - HEADROOM):
            continue                                          # keep the spare capacity
        score = rtt_ms.get(z.name, 999)
        if z.name in meeting_zones:
            score -= SAME_MEETING_MS                          # avoids a new cascade link
        if score < best_score:
            best, best_score = z.name, score
    if best is None:
        raise RuntimeError("no zone with capacity: shed to the next region tier")
    return best

Deep Dive 5: Signalling and Meeting State (Week 5 Concepts: Leader Election, Fencing)#

This builds on Week 5, Day 5 · Leader Election.

What is published: after a wave of uninvited guests disrupting meetings in early 2020, Zoom turned on waiting rooms and passcodes by default from 5 April 2020 for many accounts, and in Zoom 5.0 (April 2020) grouped the host's security controls behind one icon. The control plane in this deep dive is a generic design for those features.

%% title: A participant's life in a meeting
stateDiagram-v2
  [*] --> JOINING
  JOINING --> WAITING: waiting room on
  JOINING --> IN_MEETING: admitted by policy
  WAITING --> IN_MEETING: host admits
  WAITING --> [*]: host denies
  IN_MEETING --> RECONNECTING: network change
  RECONNECTING --> IN_MEETING: same session within 60 s
  RECONNECTING --> [*]: timed out
  IN_MEETING --> [*]: leaves or removed
  class IN_MEETING ok
  class WAITING queue

• The waiting room holds signalling only. A waiting participant has a socket to the controller but no media token, so they see and hear nothing; admitting them issues the token.
• Media tokens are short-lived and scoped: meeting, participant, role, expiry of about a minute, signed by the control plane. The MMR checks the signature and never calls back on the hot path. Removing someone revokes by telling their MMR to drop the session now.
• The top-of-hour herd hits the control plane, not the media plane: ~60,000 joins a second. Spread controllers by meeting ID, keep join a single round trip after authentication, and let clients jitter their retries.
• The host leaves: the controller promotes a co-host, or the earliest-joined participant if the account allows; a meeting with no one left ends after a grace period.

# meeting/controller.py

"""
A meeting controller (illustrative). One owner per meeting, commands applied in
order, every change versioned and fanned out; the lease epoch fences off a stale
owner after failover.
"""


class Fenced(Exception):
    pass


class MeetingController:
    def __init__(self, meeting_id: str, epoch: int, store, bus):
        self.id, self.epoch, self.store, self.bus = meeting_id, epoch, store, bus
        self.version = 0
        self.state = {"locked": False, "waiting": [], "in": {}, "hosts": set(), "self_unmute": True}

    def apply(self, actor: str, cmd: dict) -> dict:
        if not self.store.lease_is(self.id, self.epoch):
            raise Fenced(f"epoch {self.epoch} no longer owns {self.id}")
        s, op = self.state, cmd["op"]
        if op in ("lock", "admit", "remove", "mute_all") and actor not in s["hosts"]:
            return {"ok": False, "why": "host or co-host only"}
        if op == "lock":
            s["locked"] = True
        elif op == "admit":
            if s["locked"]:
                return {"ok": False, "why": "meeting is locked"}
            s["waiting"].remove(cmd["who"])
            s["in"][cmd["who"]] = {"muted": True}
        elif op == "mute_all":
            for p in s["in"].values():
                p["muted"] = True
        elif op == "unmute_self":
            if not s["self_unmute"]:
                return {"ok": False, "why": "host has disabled self-unmute"}
            s["in"][actor]["muted"] = False
        self.version += 1
        self.bus.publish(self.id, {"v": self.version, "by": actor, **cmd})   # to every client, in order
        return {"ok": True, "v": self.version}

Deep Dive 6: Cloud Recording and End-to-End Encryption (Week 9 Concepts: Security Architecture)#

This builds on Week 9, Day 5 · Security Architecture.

• The recorder is a participant. A recording worker joins the meeting like a silent client, subscribes to the active speaker, the screen share and all audio, decodes and composes a file, and uploads it to object storage in chunks as the meeting goes. If it crashes, a new one joins and the file has a gap, not a corruption.
• Ingest, then process. The live composite is written once; transcoding to other sizes, thumbnails and transcripts run afterwards from a queue. ~360 TB a day (Phase 2's estimate) is a storage-tier and lifecycle problem: move older recordings to colder storage, delete on the account's retention policy.
• Standard encryption protects media on every hop: Zoom 5.0 added 256-bit AES-GCM in April 2020, enabled system-wide from 30 May 2020. But the keys come from Zoom's servers, so the service could in principle decrypt.
• End-to-end encryption moves key generation to the clients. Zoom's design, shipped as a preview in October 2020, has the meeting's host generate keys and distribute them with public-key cryptography; Zoom's servers become relays that "do not have the necessary decryption key". The tradeoff Zoom listed: up to 200 participants, and no cloud recording, live transcription, breakout rooms or polling.
• Why an SFU makes E2EE possible at all: a forwarding server only needs the headers, which stay readable; the payload can be encrypted end to end. A mixing MCU must decode, so it could never be end-to-end encrypted.

Kavya's Monday Morning#

Edge Cases#

Monitoring Dashboard#

SLOs for the Platform#

The interviewer leans back. "One more."

Summary: Concepts Applied from 10-Week Course#

Why Zoom Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Media architecture:

Adaptation and transport:

Placement, control and operations:

Sources#

Zoom and its suppliers:

• Blog

  Eric S. Yuan — A Message to Our Users (about 10 million daily meeting participants at the end of December 2019; more than 200 million in March 2020; 90,000 schools in 20 countries), Zoom, 1 April 2020: https://www.zoom.com/en/blog/a-message-to-our-users/

• Read

  SiliconANGLE — Zoom retracts claim of 300M daily active users (the 22 April figure described as users, corrected to 300 million daily meeting participants; participants counted per meeting), 30 April 2020: https://siliconangle.com/2020/04/30/zoom-retracts-claim-300m-daily-active-users-rivals-narrow-gap/

• Blog

  Zoom — Zoom Hits Milestone on 90-Day Security Plan, Releases Zoom 5.0 (256-bit AES-GCM; system-wide enablement on 30 May), 22 April 2020: https://www.zoom.com/en/blog/zoom-hits-milestone-on-90-day-security-plan-releases-zoom-5-0/

• Talk

  Zoom Q3 FY2021 earnings call, Kelly Steckelberg (annualized run rate of 3.5 trillion meeting minutes at the end of the quarter), 30 November 2020: https://www.fool.com/earnings/call-transcripts/2020/12/01/zoom-video-communications-zm-q3-2021-earnings-call

• Talk

  Zoom Video Communications — Form 10-K for the fiscal year ended 31 January 2021 (a multimedia router on commodity hardware and globally distributed cloud infrastructure; AWS and Oracle Cloud): https://www.sec.gov/Archives/edgar/data/1585521/000158552121000048/zm-20210131.htm

• Read

  Zoom — Zoom: Architected for Reliability (MMRs grouped in meeting zones; zone controllers and a global cloud controller; geolocation steering; UDP with fallback to TCP/TLS including HTTPS/443; multiple simultaneous streams and layer selection; 50% excess capacity; 99.9% availability; RTO goal under five minutes): https://library.zoom.com/admin-corner/architecture-and-design/zoom-architected-for-reliability

• Blog

  Zoom Support — Zoom network firewall or proxy server settings (UDP 8801–8810; TCP 443, 8801, 8802): https://support.zoom.us/hc/en-us/articles/201362683

• Blog

  Max Krohn — Zoom Rolling Out End-to-End Encryption Offering (host-generated keys distributed with public-key cryptography; up to 200 participants; no cloud recording, live transcription, breakout rooms, polling or private chat), Zoom, 14 October 2020: https://www.zoom.com/en/blog/zoom-rolling-out-end-to-end-encryption-offering/

• Talk

  Zoom — Zoom unveils 1 million attendee capability for Zoom Webinars (up to 1,000 interactive video panelists), 19 August 2024: https://news.zoom.com/zoom-unveils-1-million-attendee-zoom-webinars/

• Read

  AWS — AWS and Zoom Extend Strategic Relationship (from 10 million daily meeting participants in December 2019 to more than 300 million a day regularly since April 2020; thousands of EC2 instances added in a day), 30 November 2020: https://press.aboutamazon.com/2020/11/aws-and-zoom-extend-strategic-relationship

Reporting:

• Docs

  Data Center Frontier — Inside Zoom's Infrastructure: Scaling Up Massively With Colo and Cloud (17 data centres; nine Equinix sites as of November 2019; seven petabytes a day through Oracle Cloud Infrastructure), 22 May 2020: https://datacenterfrontier.com/inside-zooms-infrastructure-scaling-up-massively-with-colo-and-cloud/

• Talk

  KrASIA — Video conferencing platform Zoom to strengthen its presence in India (2 of 17 data centres in Mumbai and Hyderabad, 4× capacity in four months; 6,700% more free sign-ups in India from January to April 2020), 22 July 2020: https://kr-asia.com/video-conferencing-platform-zoom-to-strengthen-its-presence-in-india

• Book

  The Register — Zoom's end-to-end encryption preview (keys generated by participants' machines; servers without the decryption key; 200 participants), 27 October 2020: https://www.theregister.com/2020/10/27/zoom_endtoend_encryption/

• Book

  Engadget — Zoom will enable waiting rooms by default to keep trolls out (waiting rooms and passwords from 5 April 2020), 4 April 2020: https://www.engadget.com/2020-04-04-zoom-waiting-rooms-default.html

• Read

  Philipp Hancke — How Zoom's web client avoids using WebRTC (WebSockets and WebAssembly in 2018; WebRTC data channels from 2019), webrtcHacks, updated 8 September 2019: https://webrtchacks.com/zoom-avoids-using-webrtc/

Standards:

• Read

  ITU-T G.114 — One-way transmission time (most applications not significantly affected below about 150 ms; 400 ms as the planning limit), May 2003: https://www.itu.int/rec/T-REC-G.114

• Docs

  Stefan Holmer, Henrik Lundin, Gaetano Carlucci, Luca De Cicco, Saverio Mascolo — A Google Congestion Control Algorithm for Real-Time Communication, draft-ietf-rmcat-gcc-02 (loss-based rules at 2% and 10%; delay-gradient over-use detection), July 2016: https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02

• Docs

  RFC 4585 — Extended RTP Profile for RTCP-Based Feedback (NACK, PLI): https://www.rfc-editor.org/rfc/rfc4585

• Docs

  RFC 5109 — RTP Payload Format for Generic Forward Error Correction: https://www.rfc-editor.org/rfc/rfc5109

• Docs

  RFC 6716 — Definition of the Opus Audio Codec (in-band FEC): https://www.rfc-editor.org/rfc/rfc6716

• Talk

  RFC 6190 — RTP Payload Format for Scalable Video Coding: https://www.rfc-editor.org/rfc/rfc6190

• Docs

  RFC 8853 — Using Simulcast in SDP and RTP Sessions: https://www.rfc-editor.org/rfc/rfc8853

Design note: the architecture, code, thresholds, bitrates, server counts and dashboards here are an illustrative design for the problem, not a description of Zoom's internal systems beyond what Zoom has published. Estimates in Phase 2 are labelled as assumptions or estimates; the round-trip times between continents are estimates.

Further Reading#

Related systems to study:

• Read

  WhatsApp (Bonus Problem 2 in this series): signalling, presence and end-to-end encryption for messages

• Talk

  Netflix (Bonus Problem 6): adaptive bitrate for video on demand, where seconds of buffer are allowed

• Talk

  YouTube (Bonus Problem 19): transcoding ladders and why live and on-demand differ

• Docs

  Figma (Bonus Problem 15): one server per document, ordered changes and cursors in real time

Papers and specifications:

• Paper

  Zoom — E2E Encryption for Zoom Meetings (the cryptographic design whitepaper, first published May 2020): https://github.com/zoom/zoom-e2e-whitepaper

• Docs

  RFC 8888 — RTP Control Protocol Feedback for Congestion Control

Foundational concepts (this course):

• Read

  Week 1, Day 1 — Partitioning Deep-Dive

• Read

  Week 2, Day 1 — Timeout Hell

• Read

  Week 3, Day 3 — Backpressure and Flow Control

• Read

  Week 5, Day 5 — Leader Election

• Read

  Week 9, Day 3 — Data Residency

• Read

  Week 9, Day 5 — Security Architecture

• Read

  Week 10, Day 4 — Capacity Planning

"Late is the same as lost; so forward, don't mix, and give every receiver the best picture its network can carry."