A model of several terabytes, updated every minute

transcript

In ByteDance's Monolith paper, models are several terabytes. And serving learns what changed every minute. Without stopping.

Copy the whole model each minute? Terabytes, every minute. And swapping it in doubles the memory you need. Here's the trick. In a short window, only a few IDs change. Push just those rows. Serving overwrites them in place.

The trainer keeps a set of every key it touches. Every minute, only those rows ship. The paper's example: 100,000 IDs a minute, at 1,024 dimensions. About 400 megabytes. Dense weights move slower. They ship once a day.

But wait. Minute-old embeddings beside day-old dense weights? Dense weights move slowly. An embedding can change a lot in a minute. Monolith reports no conspicuous loss.

Servers die. Say 0.01 percent a day each. Across 1,000 servers. One fails about every ten days. With daily snapshots, it loses a day of updates at most.

Don't ship the model. Ship what changed.

Full breakdown in the lecture.