Revise · W11 · Day 1LLM Serving: Tokens, Latency and Cost
30:00
Key ideaDay 1 · 1/10
Size it in tokens, not requests
A model call's work, price and rate limits scale with tokens in and out. 900 turns a minute sounds small; at ~3,835 input tokens a turn it's 3.45M input tokens a minute.
← → flip · space next / show answer · S shuffle · esc exit1 / 59