Why one chat reply shares a GPU with 224 others

transcript

Give one chat reply four top GPUs. The compute sits 99 point 7 percent idle. So your reply shares them. With about 224 strangers.

Take an open 70 billion parameter model. Every token reads all 140 gigabytes of weights. That takes about 10 point 5 milliseconds. So one user gets about 95 tokens a second. Sounds fine.

But here's the catch. Decode does about one calculation per byte it reads. An H100 can do about 295. So the compute is 0 point 3 percent busy. It's just waiting on memory.

So here's the trick. Read the weights once per step. And use them for every reply in the batch.

Alone, you got about 95 tokens a second. With 225 replies in the batch, you get about 46. Half. But the replica now writes about 10,500 a second. 110 times more.

And 46 tokens a second? People read about 5 point 6. You'll never notice you're sharing.

Decode is memory-bound. So one step must serve hundreds of replies. Batching is the whole economics. The craft is making it invisible.

Full breakdown in the lecture.