~learn.

Google Docs Collaboration, the whole design.

Up to 100 people typing into the same text at once, kept identical by one server per document that transforms late changes past newer revisions and appends each one to a revision log that is the document, its fence and its history.

6 diagrams · 8 sections · revise in ~10 min · Download PDF prints it in the light theme, diagrams included

SQ-20 · Google Docs Collaboration · design sheethimanshukukreja.in/learn/sheets/bonus-20-google-docs-collaboration

At a glance#

Number Value What it drives
People in one file 100 at once past 100 only the owner and some editors can edit
Document size ≤ 1.02 million characters one document always fits one server's memory
Named versions 40 per document labels on revision numbers, never coarsened away
One busy document 40 typists → 200 changes / s in · ~20,000 messages / s out fan-out grows with the square of the room
Fleet (assumed) 25M peak sessions · 20M open docs · 15M changes / s Google hasn't published Docs usage
Memory ~10 TB on ~270 servers ~0.5 MB per open document (assumed)
Revision log ~41 TB / day before replicas snapshots every 1,000 revisions; coarsen after 30 days
Latency own typing next frame · others p95 < 200 ms optimistic local apply, one change in flight

Architecture#

DiagramCollaborative editing on one page: one owner per document
%% title: Collaborative editing on one page: one owner per document
flowchart TD
  WEB@{ icon: "sd:browser", label: "Browser + mobile editors<br/>local text · one change in flight · pending" } ==>|"changes, cursors"| GW@{ icon: "sd:gateway", label: "Session gateways<br/>WebSocket · route by document" }
  GW -.->|"role? owner server?"| CTL@{ icon: "sd:lock", label: "Permissions + router<br/>role per file · document → server" }
  GW ==> DS["Document server · one owner per document<br/>transform past newer revisions<br/>number · append · ack · broadcast"]:::hot
  DS --> PR@{ icon: "tech:redis", label: "Redis · presence<br/>cursor = position + revision · TTL" }
  DS ==>|"append N+1 if last = N"| LOG@{ icon: "sd:log", label: "Revision log<br/>every change · author · seq" }
  DS -.-> SNAP@{ icon: "sd:object-storage", label: "Snapshots<br/>every 1,000 revisions" }
  LOG --> VH["Version history<br/>groups · 40 named · restore"]:::service
  LOG --> CF@{ icon: "tech:kafka", label: "Kafka · change feed" }
  CF --> IDX@{ icon: "sd:search", label: "Search · comments · notifications" }
  class WEB client
  class GW edge
  class CTL service
  class PR cache
  class LOG,SNAP store
  class CF queue
  class IDX service
DiagramInside the document server: one change
%% title: Inside the document server: one change
flowchart TD
  IN(["Change · client c7, seq 41<br/>made against revision 811"]):::client --> DUP{"seq 41 already applied<br/>for c7?"}:::service
  DUP -->|"yes"| ACK(["Ack again<br/>a resend after reconnect"]):::ok
  DUP -->|"no"| ROLE{"Can this session edit?"}:::service
  ROLE -->|"no"| REJ(["Reject VIEW_ONLY<br/>editor drops its changes"]):::bad
  ROLE -->|"yes"| TX["Transform past 812 … latest<br/>Insert 40 → Insert 24"]:::hot
  TX --> AP{"Append revision N+1<br/>if the log ends at N?"}:::core
  AP -->|"fails"| STALE(["Stale owner · stop serving"]):::bad
  AP -->|"ok"| OUT["Ack to author · broadcast to others<br/>editors shift cursors locally"]:::core

Key flows#

DiagramTwo edits to one sentence, merged by one server
%% title: Two edits to one sentence, merged by one server
sequenceDiagram
  autonumber
  participant A as Ananya
  participant S as Document server
  participant L as Revision log
  participant K as Kabir
  Note over A,K: both at revision 811
  K->>S: rev 811 · Delete 20-35
  A->>S: rev 811 · Insert 40 "the"
  S->>L: Append 812 if last is 811
  S-->>K: ack 812
  S-)A: 812 · Delete 20-35
  A->>A: Transform past her pending insert, apply
  S->>S: Transform her insert past 812 · now 24
  S->>L: Append 813 if last is 812
  S-->>A: ack 813
  S-)K: 813 · Insert 24 "the"
  Note over A,K: both screens show the same sentence
DiagramThree days offline: merge by composition
%% title: Three days offline: merge by composition
sequenceDiagram
  autonumber
  participant K as Kabir's editor
  participant W as Background worker
  participant S as Document server
  participant L as Revision log
  K->>S: Hello · base revision 112,000 · 2,400 changes held
  S->>W: Compose revisions 112,001 to 172,000
  W->>L: Read 60,000 revisions
  alt base still inside the fine-grained window
    W-->>K: One composed change, bounded by the document
    K->>K: Save a version · compose his 2,400 · transform one vs one
    K->>S: One change against 172,000
    S->>S: Transform past the few newer revisions · append · ack
  else base older than the window
    W-->>K: Three-way merge from his base snapshot
  end

Data model#

DiagramCore entities
%% title: Core entities
classDiagram
  class Document {
    +docId : string
    +rev : int · latest revision
    +text : string · ≤ 1.02M characters
    +owner : server · from the router
  }
  class Revision {
    +docId : string
    +rev : int · N+1 only if last is N
    +ops : insert / delete / style list
    +author : user
    +clientSeq : client and seq · dedupe
    +at : timestamp
  }
  class Snapshot {
    +docId : string
    +rev : int · every 1,000
    +content : text and styles
  }
  class NamedVersion {
    +docId : string
    +rev : int
    +name : string · up to 40
  }
  class Presence {
    +session : string
    +anchor : int
    +head : int
    +rev : int · position is relative to it
    +ttl : 30 s
  }
  class CommentAnchor {
    +start : int · sticks left
    +end : int
    +orphaned : bool
  }
  Document "1" --> "many" Revision : history
  Document "1" --> "many" Snapshot : opens fast from
  Document "1" --> "many" NamedVersion : labelled
  Document "1" --> "many" Presence : live cursors
  Document "1" --> "many" CommentAnchor : comments

Lifecycle#

DiagramOne local change, from keystroke to the log
%% title: One local change, from keystroke to the log
stateDiagram-v2
  [*] --> PENDING: typed, applied locally
  PENDING --> PENDING: more typing, composed
  PENDING --> IN_FLIGHT: nothing else in flight, sent
  IN_FLIGHT --> IN_FLIGHT: remote revision, transformed
  IN_FLIGHT --> IN_FLIGHT: reconnect, resent with same seq
  IN_FLIGHT --> COMMITTED: ack with revision number
  IN_FLIGHT --> DROPPED: rejected, view only
  COMMITTED --> [*]
  DROPPED --> [*]
  class COMMITTED ok
  class DROPPED bad
  class IN_FLIGHT queue

Decisions#

Decision Chose Over Because
Merge model Server-ordered OT Text CRDT · last-writer-wins server exists anyway; only TP1 needed; no per-character IDs
Changes in flight One per client, the rest composed Stream every keystroke server keeps one history, never a client's branch
Durability Append, then ack (group commit) Async journal an acknowledged change is never lost
Fencing Conditional append on revision N Separate lock service the log's own sequence stops a stale owner
Cursors Position + revision, shifted locally Broadcast after every edit others' typing moves cursors with no messages
Restore Diff appended as a new revision Truncate the log history intact; a restore can be undone
Long offline merge Compose both sides, transform once Change-by-change transforms 144M transforms become one, off the hot path

When it breaks#

Failure What users see Contain it with
Document server crashes a short pause, no lost typing snapshot + log tail; resend in-flight change, dedupe by (client, seq)
Stale owner wakes up nothing its conditional append fails; it stops serving
101st person opens the doc live read-only view limit of 100; publish to the web for more
Editor demoted mid-sentence changes dropped, offer to copy next change rejected VIEW_ONLY; reload read-only
A copy diverges (OT bug) wrong text until reload text hash every 500 revisions; reload and investigate
Snapshotter stuck slow opens and failovers alert on snapshot lag; durability unaffected

Say it in the interview#