LLM serving: tokens, latency and cost

MegaMart Assist's launch evening, and the model call underneath it: tokens and tokenizers, the context window and a conversation that grows every turn, prefill and decode, time to first token and time per output token, static vs continuous batching, the KV cache and why tokens resident set concurrency, vendor API or self-hosted vLLM, streaming through your own stack (buffering proxies, heartbeats, errors after the 200), provider rate limits and spend caps, an LLM gateway that admits by tokens, and the cost arithmetic that makes input three-quarters of the bill.