At a glance#
| Number | Value | What it drives |
|---|---|---|
| Users → open files | 13M+ MAU → 1.5M peak sessions · 600,000 files | 40 % daily, 3 h, 2× peak, 2.5 sessions a file (assumptions) |
| One file's ceiling | 500 people · 200 editing · 200 cursors | the biggest review, not the average file, sizes presence |
| Changes | 2.2B+ / day ≈ 25,000 / s | design peak 100,000/s; ordering is hard, carrying is not |
| Cursors | ~3.6M updates / s, 36× edits | own lossy path; one 500-person file is ~2M entries/s, 256 Mbps naive |
| Documents in memory | ~12 TB on ~175 servers | ~20 MB mean file; memory runs out before CPU |
| Saved | 95 % within 600 ms | journal batch every 0.5 s; ~432 GB a day |
| Latency | own edit next frame · others p95 < 150 ms | optimistic local apply; one owner echoes |
Architecture#
DiagramMultiplayer on one page: one owner per file
%% title: Multiplayer on one page: one owner per file
flowchart TD
ED@{ icon: "sd:browser", label: "Browser + desktop editors<br/>local copy of the file · outbox" } ==>|"changes, cursors"| GW@{ icon: "tech:websocket", label: "WebSocket gateways · ~60<br/>TLS · session · route by file" }
ED --> API@{ icon: "sd:cdn", label: "HTTP API + CDN<br/>images, fonts by content hash" }
API --> META@{ icon: "tech:postgresql", label: "LiveGraph over Postgres<br/>comments · file browser" }
GW -.->|"who owns file F?"| RT@{ icon: "sd:key", label: "Router + lock table<br/>file → owner, lock UUID<br/>role per file · revocations" }
GW ==> MP["Multiplayer server · one process per file<br/>whole file in memory<br/>checks + sequences every change"]:::hot
RT -.->|"revoke edit"| MP
MP --> PR@{ icon: "sd:cache", label: "Presence<br/>latest cursor per session · memory only" }
MP ==>|"~0.5 s batches, if lock matches"| JR@{ icon: "tech:dynamodb", label: "Journal · DynamoDB<br/>per-file sequence numbers" }
MP --> CK@{ icon: "tech:s3", label: "Checkpoints · S3<br/>compressed file · every 10 min" }
CK --> VH["Version history<br/>autosave every 30 min · named · branches"]:::service
class ED client
class GW,API edge
class META,JR,CK store
class RT store
class PR cache- One authority per file: every editor of a file connects to the same owner, whose sequence number is the only clock. No consensus runs per edit.
- The journal sets durability; checkpoints set recovery time. Cursors share the socket but never touch storage.
- Comments and the file browser run on a separate real-time system; images and fonts live outside the document by content hash.
DiagramInside the owner: one change, checked then sequenced
%% title: Inside the owner: one change, checked then sequenced
flowchart TD
IN(["Change · client 17, change_id 41<br/>button.x = 112"]):::client --> DUP{"change_id already applied<br/>for this client?"}:::service
DUP -->|"yes"| DROP(["Drop · ack again<br/>a resend after reconnect"]):::ok
DUP -->|"no"| CHK{"Can edit? Object exists?<br/>Parent change makes a cycle?"}:::service
CHK -->|"fails"| REJ(["Reject with server values<br/>client rolls back"]):::bad
CHK -->|"ok"| LWW["Write the property · last writer wins<br/>free pos if taken · seq + 1"]:::hot
LWW --> OUT["Ack to author · echo to all<br/>one frame per viewer per tick"]:::core
LWW --> BUF[["Unsaved buffer"]]:::queue
BUF -->|"every 0.5 s"| JW@{ icon: "tech:dynamodb", label: "Journal write<br/>conditional on lock UUID" }
JW --> SAVED(["Broadcast saved up to seq<br/>clients clear their outboxes"]):::ok
class JW storeKey flows#
DiagramOne nudge, Priya to Arjun and into the journal
%% title: One nudge, Priya to Arjun and into the journal sequenceDiagram autonumber participant P as Priya's editor participant S as Owner server participant J as Journal participant A as Arjun's editor P->>P: apply x = 112 now · mark unacked P->>S: change c41 · button.x = 112 S->>S: may edit? exists? · seq 9,012 S-->>P: ack c41 = seq 9,012 S-)A: seq 9,012 · button.x = 112 A->>A: apply · redraw next frame Note over P: incoming values for x ignored while unacked Note over S,J: every ~0.5 s S->>J: batch 9,001–9,040 · if lock matches J-->>S: written S-)P: saved up to 9,040 P->>P: drop c41 from the outbox
DiagramOwner crashes with Priya's changes unsaved
%% title: Owner crashes with Priya's changes unsaved
sequenceDiagram
autonumber
participant P as Priya's editor
participant R as Router + lock table
participant N as New owner
participant J as Journal + S3
participant O as Old owner (stalled)
Note over O: journal has up to 9,412 · 9,413–9,414 echoed, not saved
R->>N: own file F
N->>R: write fresh lock UUID
N->>J: load checkpoint, replay batches after it
Note over N: recovered to 9,412
P->>N: hello · old epoch, last seq 9,414
N-->>P: ahead of recovery · resync
P->>N: resend outbox · her two unsaved changes
N-->>P: sequenced again · dupes dropped by change_id
alt old owner wakes up
O->>J: write batch
J-->>O: lock mismatch · refused
O->>O: stop serving F
endDiagramDraining a host: warmed handoff of one busy file
%% title: Draining a host: warmed handoff of one busy file sequenceDiagram autonumber participant C as Editors in the file participant O as Old owner participant R as Router participant N as New owner participant J as Journal R->>N: prepare file F N->>J: load checkpoint, follow the journal Note over N: decode done before cutover R->>O: hand F to N O->>O: stop sequencing · edits wait in outboxes O->>J: flush last batch up to 9,412 · under 1 s at p99 R->>R: lock F with N's UUID · route to N O-)C: moved · reconnect jittered over 2 s C->>N: hello, last seq 9,412, then outbox N-->>C: nothing missed · acks for resends Note over C,N: a few KB each, not 40 MB × 500
Data model#
DiagramCore entities
%% title: Core entities
classDiagram
class File {
+fileKey : string · owner routing key
+seq : int · last sequenced change
+lockUuid : string · epoch, fences writes
}
class SceneObject {
+id : string · client:counter e.g. 17:204
+parent : id · a property like any other
+pos : fraction · base-95 string
+props : map · x, fill, text…
}
class Change {
+clientId : int
+changeId : int · idempotency
+obj : id
+props : map
+seq : int · set by the owner
}
class JournalBatch {
+fileKey : string
+start : int
+end : int
+changes : list · ~200 B each
}
class Checkpoint {
+fileKey : string
+seq : int
+blobKey : string · S3, ~10 % of memory size
}
class Presence {
+sessionId : string
+x : int
+y : int
+page : id · memory only, never saved
}
File "1" --> "many" SceneObject : tree of
File "1" --> "many" Checkpoint : snapshotted as
Checkpoint "1" --> "many" JournalBatch : replay after its seq
JournalBatch "1" --> "many" Change : holds
Presence "many" --> "1" SceneObject : on page- Everything keys on fileKey: one owner, one sequence, one lock. A document is a map of object id → map of property → value.
- Mean file ~20 MB in memory (~1 KB an object), ~2 MB checkpoint; journal kept 7 days ≈ 3 TB.
Lifecycle#
DiagramThe life of one local change
%% title: The life of one local change stateDiagram-v2 [*] --> LOCAL: applied at once, sent LOCAL --> SEQUENCED: ack with seq LOCAL --> ROLLED_BACK: rejected, server value LOCAL --> RESEND_PENDING: connection lost SEQUENCED --> RESEND_PENDING: owner lost it before save RESEND_PENDING --> LOCAL: reconnect, outbox resent SEQUENCED --> SAVED: saved watermark passes seq SAVED --> [*] ROLLED_BACK --> [*] class SAVED ok class ROLLED_BACK bad class RESEND_PENDING queue
Decisions#
| Decision | Chose | Over | Because |
|---|---|---|---|
| Conflict model | Per-property LWW, server order | OT · peer-to-peer CRDT | design edits are "set a property"; no transforms |
| Authority | One owner process per file, in memory | Shared DB + locks per edit | checks and sequencing are memory operations |
| Ordering clock | Owner's sequence number | Client timestamps | clock skew can't reorder edits |
| Sibling order · moves | Fractional index; reject cycles | Renumbered integer index | inserts touch one object; one cycle check |
| Durability | Journal every 0.5 s + 10-min checkpoints | Checkpoint every 30–60 s | ≤ 60 s loss → < 1 s; uploads ~27 → ~2.7 Gbps |
| Split brain | Writes conditional on lock UUID | Trust heartbeats | a stalled owner's write simply fails |
| Cursors | Coalesced, memory-only side stream | Through the change stream | 36× edits, and only the newest matters |
When it breaks#
| Failure | What users see | Contain it with |
|---|---|---|
| Owner crashes mid-review | a pause of seconds | checkpoint + journal replay; outbox resend; epoch resync |
| Stalled owner wakes up | nothing | journal write fails the lock check; it stops serving |
| Host drained | edits and cursors freeze ~1 s | warmed handoff; jittered reconnect from last seq |
| 500-person review | cursors at 10 / s, viewport only | coalesce, ~12 B entries, skip frames for slow sockets |
| Back after hours offline | own colours overwrite newer ones | autosave before merge; summary of overwritten changes |
| Cycle or deleted object | layer snaps back | reject carries server values; client rolls back |
Say it in the interview#
- "One owner per file, in memory, with a fenced journal." Ordering and validation are local work; moving a file is handing over a lock, a checkpoint and a sequence number.
- It's an LWW-Map of LWW-Maps whose timestamp is the server's sequence: different properties never conflict, the same one goes to the last arrival.
- No flicker: a client ignores incoming values for a property it has unacked; a reject carries the server's value to roll back.
- Never lose what someone saw: changes stay in the outbox until the saved watermark passes them, and per-client change ids make resends harmless.
- Journal for durability, checkpoints for recovery time: that is why checkpoints can go to every 10 minutes.
- Presence dwarfs edits: coalesce to one frame per viewer per tick, filter by viewport, never save cursors.
- Memory is the binding resource: move the biggest files with a warmed handoff, never make 500 clients reload 40 MB.