Where the first second goes before an AI assistant's first word

transcript

Your AI assistant promises first words in about a second. Where does that second actually go? Let's spend it.

Edge, auth, loading the chat. About 30 milliseconds. Rewriting a follow-up question. About 250. Searching and reranking the help articles. About 250 more. Then the model's first token. 600 to 800 milliseconds. The biggest bar.

Wait. There's a safety screen too. Another 200 milliseconds. It runs in parallel with search, and finishes inside search's time. So it adds nothing.

Add it up. About 0.9 to 1.3 seconds to the first word. Then about 20 milliseconds per token. A 250-token reply finishes about 5 seconds later.

Want under a second? You have exactly two knobs. Skip the rewrite when the question stands alone. Save 250 milliseconds. And rerank fewer candidates when it slips.

The model is only part of the second. The rest is your own pipeline. So time every step, not just the end.

Full breakdown in the lecture.