Halve the prompt, 1–5% faster. Halve the reply, ~50% faster: TTFT vs time per token

transcript

Your chatbot is slow. So you halve its prompt. It gets 1 to 5 percent faster. That's it. So where did the time go?

Every reply runs on two clocks. First, time to first token. The model reads your prompt. Then time per token. It writes the answer one token at a time.

A 250-token reply. The first token lands in about 0.8 seconds. Then 20 milliseconds a token, one after another. About 5 more seconds.

But here's the catch. Reading your prompt is the fast part. Writing the answer is where the seconds go.

OpenAI's guide: halve the prompt, maybe 1 to 5 percent faster. Halve the output instead, and you cut about half the latency. Then stream it, so first words show up in under a second.

Want a faster chatbot? Don't start by trimming the prompt. Shorten the answer, and stream it. Latency is an output problem.

Full breakdown in the lecture.