One sentence added to a 100 MB file. Why sync re-uploads all of it.

transcript

You add one sentence to a 100 megabyte file. Your sync app re-uploads all 100 megabytes. Here's why.

Upload the whole file on every save? One paragraph costs 100 megabytes. So split it into 4 megabyte chunks. 25 of them, each with its own hash. Edit a paragraph in place. One hash changes. Upload 4 megabytes, not 100.

But here's the catch. Don't edit in place. Insert. One new sentence, at the very beginning of the file.

The new bytes go in at the start. So every byte after them shifts along. The cuts stay every 4 megabytes. Now each one cuts different content. All 25 hashes change. The server is missing every chunk.

Fixed chunks are great for appends and in-place edits. Bad for insertions. So cut by content. A rolling hash picks boundaries where the bytes match a pattern. Insert a sentence, and the boundaries move with the content. Only the chunks around the insertion change. Sizes vary, say 1 to 8 megabytes.

Fixed chunks measure with a ruler. Content-defined chunks read the bytes. Cut where the content says. Then one sentence uploads only nearby chunks.

Full breakdown in the lecture.