At a glance#
| Number | Value | What it drives |
|---|---|---|
| People in one file | 100 at once | past 100 only the owner and some editors can edit |
| Document size | ≤ 1.02 million characters | one document always fits one server's memory |
| Named versions | 40 per document | labels on revision numbers, never coarsened away |
| One busy document | 40 typists → 200 changes / s in · ~20,000 messages / s out | fan-out grows with the square of the room |
| Fleet (assumed) | 25M peak sessions · 20M open docs · 15M changes / s | Google hasn't published Docs usage |
| Memory | ~10 TB on ~270 servers | ~0.5 MB per open document (assumed) |
| Revision log | ~41 TB / day before replicas | snapshots every 1,000 revisions; coarsen after 30 days |
| Latency | own typing next frame · others p95 < 200 ms | optimistic local apply, one change in flight |
Architecture#
DiagramCollaborative editing on one page: one owner per document
%% title: Collaborative editing on one page: one owner per document
flowchart TD
WEB@{ icon: "sd:browser", label: "Browser + mobile editors<br/>local text · one change in flight · pending" } ==>|"changes, cursors"| GW@{ icon: "sd:gateway", label: "Session gateways<br/>WebSocket · route by document" }
GW -.->|"role? owner server?"| CTL@{ icon: "sd:lock", label: "Permissions + router<br/>role per file · document → server" }
GW ==> DS["Document server · one owner per document<br/>transform past newer revisions<br/>number · append · ack · broadcast"]:::hot
DS --> PR@{ icon: "tech:redis", label: "Redis · presence<br/>cursor = position + revision · TTL" }
DS ==>|"append N+1 if last = N"| LOG@{ icon: "sd:log", label: "Revision log<br/>every change · author · seq" }
DS -.-> SNAP@{ icon: "sd:object-storage", label: "Snapshots<br/>every 1,000 revisions" }
LOG --> VH["Version history<br/>groups · 40 named · restore"]:::service
LOG --> CF@{ icon: "tech:kafka", label: "Kafka · change feed" }
CF --> IDX@{ icon: "sd:search", label: "Search · comments · notifications" }
class WEB client
class GW edge
class CTL service
class PR cache
class LOG,SNAP store
class CF queue
class IDX service- One server orders every change for its documents, so there is one history to transform against; clients never reconcile with each other.
- Append, then ack: an acknowledged change is already in the log; an unacknowledged one is still in its author's editor.
- Presence is disposable and never in the log; everything else follows the change feed instead of slowing typing.
DiagramInside the document server: one change
%% title: Inside the document server: one change
flowchart TD
IN(["Change · client c7, seq 41<br/>made against revision 811"]):::client --> DUP{"seq 41 already applied<br/>for c7?"}:::service
DUP -->|"yes"| ACK(["Ack again<br/>a resend after reconnect"]):::ok
DUP -->|"no"| ROLE{"Can this session edit?"}:::service
ROLE -->|"no"| REJ(["Reject VIEW_ONLY<br/>editor drops its changes"]):::bad
ROLE -->|"yes"| TX["Transform past 812 … latest<br/>Insert 40 → Insert 24"]:::hot
TX --> AP{"Append revision N+1<br/>if the log ends at N?"}:::core
AP -->|"fails"| STALE(["Stale owner · stop serving"]):::bad
AP -->|"ok"| OUT["Ack to author · broadcast to others<br/>editors shift cursors locally"]:::coreKey flows#
DiagramTwo edits to one sentence, merged by one server
%% title: Two edits to one sentence, merged by one server sequenceDiagram autonumber participant A as Ananya participant S as Document server participant L as Revision log participant K as Kabir Note over A,K: both at revision 811 K->>S: rev 811 · Delete 20-35 A->>S: rev 811 · Insert 40 "the" S->>L: Append 812 if last is 811 S-->>K: ack 812 S-)A: 812 · Delete 20-35 A->>A: Transform past her pending insert, apply S->>S: Transform her insert past 812 · now 24 S->>L: Append 813 if last is 812 S-->>A: ack 813 S-)K: 813 · Insert 24 "the" Note over A,K: both screens show the same sentence
DiagramThree days offline: merge by composition
%% title: Three days offline: merge by composition
sequenceDiagram
autonumber
participant K as Kabir's editor
participant W as Background worker
participant S as Document server
participant L as Revision log
K->>S: Hello · base revision 112,000 · 2,400 changes held
S->>W: Compose revisions 112,001 to 172,000
W->>L: Read 60,000 revisions
alt base still inside the fine-grained window
W-->>K: One composed change, bounded by the document
K->>K: Save a version · compose his 2,400 · transform one vs one
K->>S: One change against 172,000
S->>S: Transform past the few newer revisions · append · ack
else base older than the window
W-->>K: Three-way merge from his base snapshot
endData model#
DiagramCore entities
%% title: Core entities
classDiagram
class Document {
+docId : string
+rev : int · latest revision
+text : string · ≤ 1.02M characters
+owner : server · from the router
}
class Revision {
+docId : string
+rev : int · N+1 only if last is N
+ops : insert / delete / style list
+author : user
+clientSeq : client and seq · dedupe
+at : timestamp
}
class Snapshot {
+docId : string
+rev : int · every 1,000
+content : text and styles
}
class NamedVersion {
+docId : string
+rev : int
+name : string · up to 40
}
class Presence {
+session : string
+anchor : int
+head : int
+rev : int · position is relative to it
+ttl : 30 s
}
class CommentAnchor {
+start : int · sticks left
+end : int
+orphaned : bool
}
Document "1" --> "many" Revision : history
Document "1" --> "many" Snapshot : opens fast from
Document "1" --> "many" NamedVersion : labelled
Document "1" --> "many" Presence : live cursors
Document "1" --> "many" CommentAnchor : comments- Partition by document ID: one owner holds the text and a recent revision tail in memory (~0.5 MB average, assumed).
- A month-old report can hold 50,000 revisions (~6 MB of log) for 60 KB of text: history is often 100× the document.
Lifecycle#
DiagramOne local change, from keystroke to the log
%% title: One local change, from keystroke to the log stateDiagram-v2 [*] --> PENDING: typed, applied locally PENDING --> PENDING: more typing, composed PENDING --> IN_FLIGHT: nothing else in flight, sent IN_FLIGHT --> IN_FLIGHT: remote revision, transformed IN_FLIGHT --> IN_FLIGHT: reconnect, resent with same seq IN_FLIGHT --> COMMITTED: ack with revision number IN_FLIGHT --> DROPPED: rejected, view only COMMITTED --> [*] DROPPED --> [*] class COMMITTED ok class DROPPED bad class IN_FLIGHT queue
Decisions#
| Decision | Chose | Over | Because |
|---|---|---|---|
| Merge model | Server-ordered OT | Text CRDT · last-writer-wins | server exists anyway; only TP1 needed; no per-character IDs |
| Changes in flight | One per client, the rest composed | Stream every keystroke | server keeps one history, never a client's branch |
| Durability | Append, then ack (group commit) | Async journal | an acknowledged change is never lost |
| Fencing | Conditional append on revision N | Separate lock service | the log's own sequence stops a stale owner |
| Cursors | Position + revision, shifted locally | Broadcast after every edit | others' typing moves cursors with no messages |
| Restore | Diff appended as a new revision | Truncate the log | history intact; a restore can be undone |
| Long offline merge | Compose both sides, transform once | Change-by-change transforms | 144M transforms become one, off the hot path |
When it breaks#
| Failure | What users see | Contain it with |
|---|---|---|
| Document server crashes | a short pause, no lost typing | snapshot + log tail; resend in-flight change, dedupe by (client, seq) |
| Stale owner wakes up | nothing | its conditional append fails; it stops serving |
| 101st person opens the doc | live read-only view | limit of 100; publish to the web for more |
| Editor demoted mid-sentence | changes dropped, offer to copy | next change rejected VIEW_ONLY; reload read-only |
| A copy diverges (OT bug) | wrong text until reload | text hash every 500 revisions; reload and investigate |
| Snapshotter stuck | slow opens and failovers | alert on snapshot lag; durability unaffected |
Say it in the interview#
- "A change only means something against the text it was made on." OT rewrites it for the text that moved underneath.
- TP1 is the whole correctness story here: both orders give the same text, with one fixed tie-break for same-spot inserts.
- One change in flight per client is what lets the server keep a single history.
- The revision number orders and fences: append N+1 only if the log ends at N.
- Cursors are positions, so they transform too; typing elsewhere needs no cursor messages.
- History is the log: snapshot plus replay to show any moment, restore as a new revision.
- Long offline sessions compose: 2,400 × 60,000 transforms become one transform of two bounded changes.