"Design Dropbox" looks like file storage, and storage is the least interesting part: object stores already solve it. The interview is about sync. A 100 MB file changes by one paragraph; what do you upload? A laptop edits a file offline while the phone edits the same file on a train; what does each device end up with? Another device must learn about a change within seconds, without a billion devices polling every second.
Design the backend and the sync protocol for a Dropbox or Google Drive style product: a desktop and mobile client keeps a local folder in sync with the cloud and with other devices.
In scope
- Upload, download and sync of files across devices
- Shared folders
- Version history and conflicts
A Dropbox-style sync service keeps a folder identical across devices and shared users, with version history, offline edits and no silently lost edit. At an assumed 50M users that is about 100 PB of files, ~1,160 commits a second, ~9–10 Gbit/s of uploads and ~50 TB of metadata. The client hashes 4 MB chunks, asks which are missing, uploads only those to S3 through pre-signed URLs, then commits a new revision against its base revision; the metadata service appends to the namespace's journal and a notification service wakes other devices, which fetch changes since their cursor and download only new chunks. The deep dives compare fixed and content-defined chunking, limit deduplication to avoid leaks, and turn concurrent edits into a 409 and a conflicted copy; failures cover partial uploads, garbage chunks and huge shared folders.
Phase 1: Requirements (3 minutes)#
"How large can files be? Do clients work offline? Is collaborative real-time editing in scope, like Google Docs, or whole-file sync?"
"Files up to 50 GB. Yes, offline is normal. Whole-file sync only; real-time co-editing is a different product."
Functional requirements#
- Add, edit, rename, move and delete files and folders on any device; other devices converge to the same state
- Share a folder with other users, with read or edit permission
- Version history: restore an earlier version of a file (say, 30 days)
- Offline edits sync when the device reconnects
- Conflicts are surfaced, never silently resolved by discarding an edit
Non-functional requirements#
- Durable: once a file is reported as synced, it is never lost
- Fast propagation: a change appears on other online devices within seconds
- Efficient: upload and download only what changed; don't store the same bytes twice
- Consistent metadata: every device agrees on which version of a file is current
- Scale: tens of millions of users, petabytes of data, files from 1 KB to 50 GB
Phase 2: Estimation (3 minutes)#
- Storage: assume 50M users × 2 GB average = ~100 PB of logical data, before deduplication and before the storage system's own redundancy
- Changes: assume each of 10M DAU changes 10 files a day: 100M changes a day ≈ ~1,160 commits/s
- Upload volume: if a change uploads about 1 MB of new chunks on average, that's ~100 TB a day ≈ ~1.2 GB/s, roughly 9–10 Gbit/s sustained
- Metadata: 2 GB per user at ~1 MB per file is ~2,000 files per user, ~100B file entries in total; at ~500 bytes each, ~50 TB of metadata, sharded
- Chunking: with 4 MB chunks, a 100 MB file is 25 chunks; editing one paragraph re-uploads 1 chunk (4 MB), not 100 MB
"Two very different stores fall out of this. Blobs are huge, immutable and content-addressed: perfect for an object store. Metadata is small, constantly changing, and must be strongly consistent: a sharded relational database."
Phase 3: API (2 minutes)#
# 1 · tell the server which chunks the new version is made of
POST /v1/uploads
{ "namespace": "ns_alice", "path": "/thesis.docx", "size": 104857600,
"chunks": ["sha256:a1..", "sha256:b2..", "...25 hashes"] }
→ 200 { "missing": [7], "upload_urls": { "7": "https://blocks.../a7..?sig=..." } }
# 2 · upload only the missing chunks, straight to the block store
PUT https://blocks.../a7..?sig=... (4 MB of bytes)
# 3 · commit the new version, saying which version you edited
POST /v1/files/commit
{ "namespace": "ns_alice", "path": "/thesis.docx", "chunks": [...], "base_revision": 7 }
→ 200 { "revision": 8, "cursor": "j_1042" } or 409 { "conflict": true, "current_revision": 8 }
# 4 · other devices: what changed since my cursor? (long poll)
GET /v1/changes?namespace=ns_alice&cursor=j_1041
→ 200 { "entries": [ { "path": "/thesis.docx", "revision": 8, "chunks": [...] } ], "cursor": "j_1042" }Phase 4: High-level design (4 minutes)#
%% title: File sync: blocks, metadata, notifications
flowchart TD
L@{ icon: "sd:laptop", label: "Desktop client<br/>watcher, chunker, local index" } -->|"which chunks are missing? / commit"| MS["Metadata service<br/>files, revisions, journal"]:::core
L -->|"upload missing chunks<br/>(pre-signed URLs)"| BS["Block service"]:::service
BS --> S3@{ icon: "tech:s3", label: "S3 · blocks<br/>key = SHA-256 of content" }
MS --> DB@{ icon: "tech:mysql", label: "MySQL · sharded by namespace<br/>files, revisions, journal" }
MS -->|"namespace changed"| NS["Notification service<br/>long-poll connections"]:::service
NS -->|"wake up: changes since your cursor"| P@{ icon: "sd:phone", label: "Phone client" }
P -->|"GET /changes, then missing chunks"| MS
P --> S3
class L,P client
class S3,DB store- Client: watches the folder, splits changed files into chunks, hashes each chunk (SHA-256), and keeps a local index of what it has synced (path, revision, chunk hashes) so it can work out what changed after a restart or offline period.
- Block service and S3: chunks are immutable and stored under their own hash. The same chunk is stored once, however many files or versions use it. Uploads and downloads go straight to the store through pre-signed URLs, so file bytes never pass through the metadata servers.
- Metadata service: owns the truth about files: which revision is current and which chunks it consists of. Every change appends an entry to the namespace's journal; a device's position in the journal is its cursor.
- Notification service: holds a long-poll (or WebSocket) connection per device and wakes it when its namespace's journal grows. The device then asks the metadata service for changes since its cursor.
%% title: Alice edits on her laptop; her phone syncs sequenceDiagram autonumber participant L as Laptop participant M as Metadata participant B as Blocks (S3) participant N as Notify participant P as Phone L->>M: chunks of thesis.docx (25 hashes) M-->>L: missing: chunk 7 L->>B: PUT chunk 7 L->>M: commit, base revision 7 M-->>L: revision 8, journal j_1042 M->>N: namespace ns_alice changed N-->>P: wake up P->>M: changes since j_1041 M-->>P: thesis.docx revision 8, chunk list P->>B: GET chunk 7 (has the other 24)
Phase 5: Deep dives (5 minutes)#
Deep dive 1: Chunks, hashes and deduplication#
"A user inserts one sentence at the very beginning of a 100 MB file. With fixed 4 MB chunks, what gets uploaded?"
"Everything. Inserting bytes at the start shifts every later byte, so every fixed-size chunk boundary now cuts different content, and all 25 hashes change. Fixed chunks are great for appends and in-place edits, and bad for insertions."
Fixed-size chunks
- Cut every 4 MB
- Simple and fast; ideal for in-place edits and appends
- An insertion near the start changes every chunk after it
Content-defined chunks
- A rolling hash over the bytes picks boundaries where the content matches a pattern
- Boundaries move with the content, so an insertion changes only the chunks around it
- Chunk sizes vary around a target (for example 1–8 MB, averaging 4 MB)
- Hash every chunk on the client.
- Ask before sending: the server compares the hashes with chunks it already stores and returns only the missing ones.
- Upload missing chunks directly to the block store. Each chunk is keyed by its hash, so retrying an upload is naturally idempotent.
- Commit last: the metadata service accepts a commit only if every chunk it lists exists. A file's metadata never points at a chunk that isn't there.
If the server skips uploading any chunk it already has from anyone, a client can learn whether some other user stores a given file just by asking "do you have this hash?" That leaks information. Common answers: deduplicate only within a user or a shared namespace, or require proof that the client actually holds the bytes before skipping the upload.
Deep dive 2: The journal, cursors and conflicts#
"Alice edits thesis.docx offline on her laptop. Meanwhile her phone saves a different edit to the same file. The laptop reconnects. What happens?"
- Every commit names its base: the laptop's commit says "this new version is based on revision 7".
- The server checks: if the current revision is still 7, the commit becomes revision 8. That check and the journal append happen in one transaction on the namespace's shard.
- The phone got there first: the phone's edit already became revision 8. The laptop's commit says base 7, so the server answers 409 Conflict.
- Keep both: the laptop saves its version alongside as
thesis (Alice's laptop's conflicted copy).docxand syncs it as a new file. No edit is lost, and the user decides which to keep. This is the behaviour many Dropbox users know as a "conflicted copy".
Each namespace (a user's root folder, or a shared folder) has an append-only journal of changes, numbered in order. A device stores the position it has applied up to. "Give me everything after j_1041" is all it needs after a few seconds or a week offline, renames and deletes included, and replaying the same entries twice is harmless because each entry sets a path to a specific revision.
Chunks and conflicts
Q1/2 A user edits one paragraph in the middle of a 100 MB file (no bytes inserted). With fixed 4 MB chunks, roughly what is uploaded?
Answer & explanation
Answer: B
Only the chunk containing the edit gets a new hash. The server reports it as missing; the other 24 are already stored.
Q2/2 The laptop commits with base revision 7, but the server is already at revision 8. What should happen?
Answer & explanation
Answer: D
Optimistic concurrency detects the conflict. Keeping both versions guarantees no edit is lost, and the user resolves it.
Phase 6: Trade-offs and failure modes (2 minutes)#
A crash in the middle of an upload
Chunks already uploaded stay in the store, unreferenced. On retry the client asks which chunks are missing and only sends the rest. The file only changes when the commit succeeds, so other devices never see a half-uploaded version.
Garbage chunks
Deleted files, expired versions and abandoned uploads leave chunks nothing references. Reference counting on every commit is fragile; a periodic mark-and-sweep that deletes chunks unreferenced for longer than a grace period (so in-flight uploads aren't collected) is safer.
A huge shared folder
A folder shared by 50,000 people is one namespace on one shard, and every change wakes 50,000 devices. Batch notifications, rate-limit the change feed, and let devices fetch lazily.
Bandwidth
Throttle background sync, prefer LAN transfer between devices on the same network where possible, and download on demand on phones instead of syncing everything.
Deleting for real
A deleted file's chunks may be shared with old versions or other files in the namespace. Delete metadata first, let garbage collection remove chunks once no reference is left, and make sure deletion requests reach backups within the retention you promise.
Phase 7: Go deeper in the course#
- An object store from the inside (durability, erasure coding, presigned URLs): Side quest: AWS S3
- Concurrent edits, last-write-wins and keeping both: Week 5 · Day 4: Conflict Resolution
- Strong metadata, eventual blobs: Week 5 · Day 1: Consistency Models
- An append-only journal with positions: Week 3 · Day 5: Audit Log System
- Commit, then notify, without losing either: Week 3 · Day 2: Transactional Outbox
- Retrying uploads and commits safely: Week 2 · Day 2: Idempotency in Practice
- Sharding metadata by namespace, and the big shared folder: Week 1 · Day 1: Partitioning Deep-Dive and Week 1 · Day 4: Hot Keys and Skew
- Deleting data that's shared and backed up: Week 9 · Day 4: Right to Deletion
- Long polling and WebSockets: Week 0 · Part 4: Networking Fundamentals
Final check#
Final check · File sync
Q1/3 100M file changes a day, each uploading about 1 MB of new chunks. Roughly what sustained upload bandwidth is that?
Answer & explanation
Answer: C
100 TB a day ÷ 86,400 s ≈ 1.2 GB/s, which is roughly 9–10 Gbit/s.
Q2/3 Why does inserting bytes near the start of a file defeat fixed-size chunks?
Answer & explanation
Answer: A
Content-defined chunking puts boundaries where the content matches a pattern, so they move with the bytes and only nearby chunks change.
Q3/3 Why must a commit be rejected if any chunk it lists isn't in the block store yet?
Answer & explanation
Answer: C
Uploading chunks first and committing last means metadata never points at missing data, so every device that sees revision 8 can fetch all of it.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.