Seven replies finish. Their GPU seats sit empty.

transcript

Eight chat replies share one GPU batch. Seven finish fast. Then their seats sit empty. For almost the whole run.

The simple way: batch eight, and run until all finish. Seven replies are 50 tokens. One reply is 2,000 tokens. And everyone waits for it. So for 97.5 percent of the batch's life, it runs one of eight.

But here's the catch. New requests can't join a running batch. So your first token waits on someone else's long reply.

The fix: schedule every step, not every batch. A reply hits its stop token? It leaves after that step. Its seat doesn't sit empty. A waiting request joins at the next step. That's Orca's iteration-level scheduling. 36.9 times FasterTransformer's throughput, on GPT-3.

Memory had the same waste. Each request reserved room for 4,096 tokens. That's about 1.3 gigabytes, held. Most replies use a fraction. In systems vLLM's authors measured, only 20 to 38 percent held real tokens. Their fix: 16-token blocks, handed out on demand, like virtual memory pages. 2 to 4 times the throughput of FasterTransformer and Orca.

Replies start at different times, and stop at unknown times. So fill every seat, every step. And hand out memory in pages.

Full breakdown in the lecture.