LLM Serving: Tokens, Latency and Cost.

MegaMart, the online retailer from Week 2's flash sale, turns on MegaMart Assist on a sale evening: a chat assistant for questions about orders, returns and refunds. (An illustrative scene: people, times and amounts are made up; each failure is a classic one, and the arithmetic uses this lesson's numbers.)

None of these is a model-quality problem. Each comes from treating a model call like an ordinary API call. It isn't: its work, duration, price and rate limits all scale with the tokens in and out.

This week builds MegaMart Assist piece by piece. Today is the bottom layer, the model call: tokens, where the seconds go, why the provider's GPUs behave as they do, streaming through your own stack, rate limits, and the cost arithmetic.

01Chapter 1: What an LLM Call Is#

1.1 The Simple Definition#

A large language model turns a sequence of tokens into the next token, again and again, in two phases:

• Prefill. It reads every input token (instructions, history, the new message) in one parallel pass.
• Decode. It writes the reply one token per step, each depending on the last, until a stop token or max_tokens.

To your service, that makes the model an unusual dependency: its duration, memory and price grow with the tokens in and out, and you don't know the output length until it's done.

Side quest 14 designs the other side of this call, the GPU fleet (Side quest 14 · ChatGPT LLM Serving). Today you are the application calling a model, behind a vendor's API or on GPUs you run.

1.2 Tokens#

A token is the unit the model reads and writes: a word, part of a word, punctuation or bytes. Google's Gemini docs give the usual rule of thumb: "a token is equivalent to about 4 characters" and "100 tokens is equal to about 60-80 English words". Three facts make such rules risky:

• Each model family has its own tokenizer. Anthropic's pricing page says Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text" than earlier Claude models (Anthropic docs, pricing). So compare cost per task, not price per token.
• Languages differ, a lot. Petrov et al. found that "the same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases", which affects cost, latency and how much fits in the context (NeurIPS 2023). That's finance's Hindi question.
• Non-text input is tokens too. Gemini counts small images as 258 tokens and audio at 32 tokens per second (Gemini API docs). Every attachment is a line on the bill.

So count with the provider's tokenizer: Anthropic's token counting endpoint is "free to use" and returns an estimate (Anthropic docs, token counting). Exact counts arrive afterwards, in each response's usage.

1.3 The Context Window#

The context window is the most tokens one call can hold, and everything counts: system prompt, messages, tool definitions, documents, plus the output and any thinking (Anthropic docs, context windows). As of October 2026, Anthropic's current models have a 1M-token window and up to 128k output tokens.

Two consequences shape the design:

• The model is stateless. Your service re-sends the history every turn, so input grows every turn.
• Bigger isn't better. Anthropic warns that "as token count grows, accuracy and recall degrade, a phenomenon known as context rot". What goes into context is a design decision (Day 3).

02Chapter 2: Where the Time Goes#

2.1 Two Phases, Two Bottlenecks#

• Prefill processes all input tokens at once: compute-bound. It sets the wait for the first token.
• Decode produces one token per sequence per step, reading the weights from GPU memory every time: "bottlenecked on how quickly we can load model parameters from device memory to compute units" (Databricks, 2023). Memory-bandwidth-bound, it sets how fast the rest arrives.

Side quest 14's Deep Dive 1 does this arithmetic for a 70B model on H100s. Here you need the formula it produces.

2.2 The Latency Formula#

Databricks defines the two numbers every LLM feature lives by (Databricks, 2023):

• Time to first token (TTFT): "how quickly users start seeing the model's output after entering their query."
• Time per output token (TPOT): the time to generate an output token for each user.
• Latency = TTFT + TPOT × (number of tokens to be generated).

NVIDIA's benchmarking docs add that TTFT "generally includes request queuing time, prefill time, and network latency", and that inter-token latency (TPOT) covers decode only: (e2e − TTFT) ÷ (output tokens − 1).

For a 250-token reply at an assumed 20 ms per token, decode takes ~5 s; TTFT is under a second. Output length dominates. OpenAI's latency guide: "cutting 50% of your output tokens may cut ~50% of your latency", while "cutting 50% of your prompt may only result in a 1–5% latency improvement".

Two things hide inside these numbers:

• Thinking. Models that reason first spend output tokens the user never sees; on Claude they're "billed as output tokens" and count toward max_tokens. LinkedIn found chain-of-thought improves quality, "But they require tokens that the member never sees, hence increasing their perceived latency" (LinkedIn Engineering, 2024).
• Load. LinkedIn also reports that "TimeToFirstToken (TTFT) & TimeBetweenTokens (TBT) increase with utilization". Chapter 3 explains why.

2.3 Measure What the User Feels#

Measure TTFT and TPOT at your edge, per feature, as percentiles; a provider's dashboard can't see your proxy or your queue:

# assist/llm_timing.py: time one streamed reply the way the customer experiences it (illustrative)
import os, time
import anthropic

client = anthropic.Anthropic()
MODEL = os.environ.get("ASSIST_MODEL", "claude-opus-5-5")

def timed_reply(system: str, messages: list[dict], max_tokens: int = 2048) -> dict:
    t0, first, parts = time.monotonic(), None, []
    with client.messages.stream(
        model=MODEL,
        max_tokens=max_tokens,                    # thinking + answer must fit in here
        system=system,
        messages=messages,
        output_config={"effort": "low"},          # chat is latency-sensitive: start low, raise if evals demand
    ) as stream:
        for text in stream.text_stream:           # visible text only: thinking happens before the first one
            if first is None:
                first = time.monotonic()
            parts.append(text)
        final = stream.get_final_message()
    end = time.monotonic()
    first = first or end
    out = final.usage.output_tokens               # billed output, thinking included
    return {
        "ttft_s": first - t0,                     # what the customer waits for: queue + prefill + thinking
        "tpot_s": (end - first) / max(out - 1, 1),  # approximate if the model thought first
        "e2e_s": end - t0,
        "input_tokens": final.usage.input_tokens,  # after the last cache breakpoint (Day 5)
        "output_tokens": out,
        "stop_reason": final.stop_reason,         # end_turn, max_tokens, refusal...
        "text": "".join(parts),
    }

Time to the first visible text, take token counts from usage, and record stop_reason: a reply cut off at max_tokens looks finished.

%% title: One MegaMart Assist turn, and where the first second goes
sequenceDiagram
  autonumber
  participant C as Customer app
  participant E as Edge proxy
  participant A as Assist service
  participant G as LLM gateway
  participant P as Model provider
  C->>E: POST reply for conversation c-77
  E->>A: forward, keep the connection open
  A->>A: load history, assemble a 3,835-token prompt
  A->>G: generate, lane interactive
  G->>G: count tokens, admit against the buckets
  G->>P: streaming request
  Note over P: queue, then prefill of 3,835 tokens
  P-->>G: 200 OK and message_start
  P-->>G: first text delta
  G-->>A: first text delta
  A-->>C: first words on screen, TTFT ends here
  loop every ~20 ms, the TPOT
    P-->>A: next delta, through the gateway
    A-->>C: delta event
  end
  P-->>G: message_delta with usage, then message_stop
  G->>G: settle buckets, emit a usage event
  A->>A: save the reply with its stop reason

03Chapter 3: Batching and the KV Cache#

You may never touch a GPU, but the provider's batching and memory decide your TPOT, your TTFT under load, your overload errors and your price.

3.1 Why Providers Batch#

If a decode step produces a token for 64 sequences at once, the weights are read once for all of them. Databricks measured one A100 serving MPT-7B with static batching, going from batch size 1 to 64: "latency increases by 4x while throughput increases by 14x" (Databricks, 2023). Batching trades a little of each user's speed for a lot of throughput, which is why your TPOT worsens when everyone else's traffic peaks.

3.2 Static vs Continuous Batching#

Anyscale saw the same at small scale: on OPT-13B on one 40 GB A100, "8x throughput over naive batching by using continuous batching", and "up to 23x" with vLLM's memory optimisations too (Anyscale, June 2023). Side quest 14's Deep Dive 2 walks through such a scheduler line by line.

3.3 The KV Cache: Why Tokens Resident Set Concurrency#

Prefill computes a key and a value vector for every input token in every layer and keeps them in GPU memory, so decode doesn't recompute the prompt each step: the KV cache. Per token it's 2 × layers × KV heads × head dimension × bytes; Databricks' example, Llama 2 70B with 16 sequences of 1,024 tokens, holds about 5 GiB in FP16 (Databricks, 2023). Three consequences:

• Concurrency is limited by tokens resident, not requests. The memory left after the weights caps how many sequences fit in the batch; a 20,000-token prompt holds the memory of five 4,000-token ones until its reply ends. It's why limits are counted in tokens, and some pricing tracks prompt length directly: Anthropic lists higher Claude Haiku 5.5 prices "for prompts over 100,000 tokens" (as of October 2026).
• Wasted memory is wasted throughput. The vLLM paper (PagedAttention) allocates KV cache in small blocks on demand, like virtual-memory pages, achieving "near-zero waste in KV cache memory" and 2-4× the throughput of FasterTransformer and Orca at the same latency (Kwon et al., SOSP 2023).
• Memory traffic is the cost. FlashAttention computes exact attention while reducing "the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM" (Dao et al., 2022).

And for Day 5: a KV cache kept between requests is a prefix cache. When two requests share their first 3,000 tokens, the second can skip that part of the prefill; providers sell this as prompt caching.

3.4 API or Self-Hosted?#

Call an API and all this is the provider's problem; self-host and it's yours. vLLM serves OpenAI-compatible Completions and Chat Completions APIs via vllm serve <model> (vLLM docs), so your code looks the same either way.

Many teams use both: an API for the hardest work, a self-hosted small model for high-volume simple calls. Either way, put a gateway between features and models (Chapter 5), so the choice can change without touching every feature.

%% title: MegaMart Assist v1: the pieces around the model call
flowchart TD
  U@{ icon: "sd:phone", label: "Customer<br/>app or browser" }
  E@{ icon: "tech:nginx", label: "Edge proxy<br/>TLS, SSE pass-through" }
  A["Assist service<br/>conversation, prompt assembly, relay"]:::core
  DB@{ icon: "tech:postgresql", label: "PostgreSQL<br/>conversations, replies, status" }
  G["LLM gateway<br/>count, admit, retry, meter"]:::core
  R@{ icon: "tech:redis", label: "Redis<br/>shared token buckets" }
  K@{ icon: "tech:kafka", label: "Kafka<br/>usage events: tokens, cost" }
  P@{ icon: "sd:cloud", label: "Model provider API<br/>RPM, ITPM, OTPM limits" }
  V@{ icon: "sd:servers", label: "Self-hosted pool, optional<br/>vLLM on GPUs" }
  M@{ icon: "sd:dashboard", label: "Dashboards<br/>TTFT, TPOT, cost per conversation" }
  U --> E --> A
  A --> DB
  A --> G
  G -.-> R
  G --> P
  G -.->|"if self-hosting"| V
  G --> K --> M
  class U client
  class E edge
  class DB store
  class R cache
  class K queue
  class P external
  class V,M service

04Chapter 4: Streaming Responses#

4.1 Why Stream#

Without streaming, the customer waits ~5.8 s; with it, they read from ~0.8 s. Streaming makes the wait useful. LinkedIn makes "all our requests stream to reduce perceived latency", and streams "within our pipeline end to end", firing downstream calls as soon as their parameters are ready (LinkedIn Engineering, 2024).

Providers stream as server-sent events (SSE): on Anthropic's API, message_start, content-block deltas, a message_delta carrying stop_reason and cumulative usage, then message_stop, with pings between (Anthropic docs, streaming). Two details matter:

• The HTTP status arrives first. Anthropic's error docs warn that "an error can occur after the API returns a 200 response", such as an overloaded_error event mid-stream, and that error handling then "doesn't follow these standard mechanisms" (Anthropic docs, errors). Automatic retries can't help once tokens flow.
• Usage comes last, in the final message_delta. Abandon a stream early and you must estimate it.

4.2 Through Your Own Stack#

Your service re-streams the reply to the customer through every proxy in between, and each hop can break it:

• Buffering proxies. nginx buffers proxied responses by default (proxy_buffering on): the 19:42 spinner. Turn it off for the stream's location, or send X-Accel-Buffering: no, which nginx honours per response (nginx docs).
• Idle timeouts. nginx closes a proxied connection if the upstream sends nothing for proxy_read_timeout, 60 s by default, measured "between two successive read operations" (nginx docs). If your relay forwards only text, a model that thinks for a minute sends nginx nothing, and it closes the connection. Send SSE comment lines as heartbeats.
• Disconnects. When the customer leaves, close the upstream request. Generating for nobody wastes capacity and, self-hosted, holds KV memory others need. Whether an API bills tokens after you disconnect depends on the provider: check, and close promptly anyway.
• Partial replies. Save every reply with how it ended (end_turn, max_tokens, refusal, cancelled, interrupted). A partial saved as complete poisons later turns, as at 20:07.

# assist/relay.py: re-stream a model reply to the browser (illustrative, FastAPI + Anthropic SDK)
import asyncio, json
import anthropic
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()
llm = anthropic.AsyncAnthropic(max_retries=2)   # retries help only before the first byte
PING_EVERY_S = 15                                 # well under every idle timeout on the path

def sse(event: str, data: dict) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"

@app.post("/assist/{conv_id}/reply")
async def reply(conv_id: str, request: Request):
    convo = await load_conversation(conv_id)      # history already trimmed to the context budget

    async def events():
        parts, status, q = [], "end_turn", asyncio.Queue()
        try:
            async with llm.messages.stream(**convo.request()) as stream:
                async def pump():                 # reads the provider; the loop below writes the client
                    try:
                        async for text in stream.text_stream:
                            await q.put(text)
                    finally:
                        await q.put(None)         # sentinel on success and on error
                task = asyncio.create_task(pump())
                while True:
                    try:
                        text = await asyncio.wait_for(q.get(), PING_EVERY_S)
                    except asyncio.TimeoutError:
                        yield ": ping\n\n"        # SSE comment: keeps idle proxies from closing us
                        continue
                    if text is None:
                        break
                    if await request.is_disconnected():
                        task.cancel()             # leaving the block closes the upstream request
                        status = "cancelled"
                        break
                    parts.append(text)
                    yield sse("delta", {"text": text})
                if status != "cancelled":
                    await task                    # re-raises an error event from mid-stream
                    final = await stream.get_final_message()
                    status = final.stop_reason    # end_turn, max_tokens, refusal...
                    await record_usage(conv_id, final.usage)
        except anthropic.APIError:
            status = "interrupted"                # the client already has a 200: tell it in-band
            yield sse("interrupted", {"kept_chars": sum(map(len, parts))})
        finally:
            await save_reply(conv_id, "".join(parts), status=status)   # never a partial marked complete
        yield sse("done", {"status": status})

    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

The queue matters: asyncio.wait_for cancels what it waits on, so waiting on the SDK's iterator directly would cancel the provider read at every heartbeat. Waiting on a queue is safe to cancel.

%% title: A streamed reply, as the assist service sees it
stateDiagram-v2
  [*] --> REQUESTED: admitted now
  [*] --> QUEUED: no budget yet
  QUEUED --> REQUESTED: buckets refilled
  QUEUED --> SHED: deadline passed
  REQUESTED --> RETRYING: 429 or 529 before any bytes
  RETRYING --> REQUESTED: after retry-after, with jitter
  REQUESTED --> STREAMING: 200 and the first delta
  STREAMING --> COMPLETE: stop reason end_turn
  STREAMING --> TRUNCATED: stop reason max_tokens
  STREAMING --> INTERRUPTED: error event or dropped connection
  STREAMING --> CANCELLED: the customer left
  INTERRUPTED --> STREAMING: continuation request
  COMPLETE --> [*]
  TRUNCATED --> [*]
  CANCELLED --> [*]
  SHED --> [*]
  class STREAMING core
  class QUEUED,RETRYING queue
  class COMPLETE ok
  class INTERRUPTED,SHED bad

4.3 When the Stream Breaks#

%% title: An error after the 200, and the continuation
sequenceDiagram
  autonumber
  participant C as Customer app
  participant A as Assist service
  participant P as Model provider
  A->>P: streaming request for turn 4
  P-->>A: 200 OK, deltas for 180 tokens
  A-->>C: 180 tokens on screen
  P-->>A: event error, overloaded_error
  A-->>C: event interrupted, partial kept and marked
  A->>A: save 180 tokens with status interrupted
  Note over A: back off with jitter before trying again
  A->>P: new request, the partial plus continue where you left off
  P-->>A: 200 OK, deltas that carry on from token 181
  A-->>C: the rest of the reply, appended
  A->>A: save the whole reply with status end_turn

The continuation is a new request: it pays for the whole prompt again, plus the partial as input.

05Chapter 5: Rate Limits and Admission Control#

5.1 How Providers Limit You#

Providers limit requests and tokens, since tokens are what consume GPU time and memory. The details differ in ways that change your design:

Sources: Anthropic docs, rate limits and errors; OpenAI docs, rate limits. As of October 2026.

Note three things. Inflating max_tokens "just in case" eats OpenAI TPM but not Anthropic OTPM. An immediate retry of a 429 fails and, on OpenAI, still counts. And Anthropic warns of acceleration limits: "a sharp increase in usage" can bring 429s, so "increase your traffic gradually". A launch from zero to full traffic at 19:30 is that pattern.

5.2 Your Own Gateway#

The provider's limiter protects the provider; yours protects your features from each other. A gateway between every feature and every model is the one place to count tokens, enforce quotas, retry and meter cost (Week 1, Day 3 · Rate Limiting at Scale). Its buckets mirror the provider's, input and output separately:

# gateway/admission.py: admit by tokens, settle from usage (illustrative, one process;
# share the buckets in Redis across gateway instances, as in Week 1 Day 3)
import time

class Bucket:
    def __init__(self, per_minute: int):
        self.capacity = self.level = float(per_minute)
        self.updated = time.monotonic()

    def refill(self) -> None:
        now = time.monotonic()
        self.level = min(self.capacity, self.level + (now - self.updated) * self.capacity / 60)
        self.updated = now

class Admission:
    def __init__(self, itpm: int, otpm: int, interactive_reserve: float = 0.25):
        self.inp, self.out = Bucket(itpm), Bucket(otpm)
        self.reserve = interactive_reserve    # share of input budget batch work may never use
        self.paused_until = 0.0

    def try_admit(self, est_input: int, expected_output: int, lane: str) -> bool:
        if time.monotonic() < self.paused_until:
            return False
        self.inp.refill(); self.out.refill()
        floor = self.inp.capacity * self.reserve if lane == "batch" else 0.0
        if self.inp.level - est_input < floor or self.out.level < expected_output:
            return False                      # caller queues until its deadline, degrades or refuses
        self.inp.level -= est_input           # input counted up front, as the provider does
        self.out.level -= expected_output     # a typical reply, not max_tokens
        return True

    def settle(self, est_input: int, expected_output: int, usage) -> None:
        counted = usage.input_tokens + (usage.cache_creation_input_tokens or 0)  # cache reads exempt here
        self.inp.level += est_input - counted
        self.out.level += expected_output - usage.output_tokens

    def on_429(self, retry_after_s: float | None) -> None:
        self.paused_until = time.monotonic() + (retry_after_s or 5.0)   # the provider's books win
        self.inp.level = min(self.inp.level, 0.0)

%% title: The gateway's admission decision for one request
flowchart TD
  IN(["Request from a feature<br/>lane, tenant, deadline"]):::client
  CNT["Count input tokens<br/>with the model's tokenizer"]:::service
  FIT{"Fits the<br/>context budget?"}
  TRIM["Trim or summarise<br/>the oldest turns"]:::service
  TEN{"Tenant and feature<br/>quota left?"}
  PROV{"Provider headroom<br/>RPM, ITPM, OTPM?"}
  SEND(["Send now<br/>stream back"]):::ok
  WAIT[["Queue by lane<br/>until the deadline"]]:::queue
  DEG(["Degrade<br/>shorter answer or smaller model"]):::service
  NO(["Refuse honestly<br/>busy, try again in N s"]):::bad
  IN --> CNT --> FIT
  FIT -->|"no"| TRIM --> TEN
  FIT -->|"yes"| TEN
  TEN -->|"no"| NO
  TEN -->|"yes"| PROV
  PROV -->|"yes"| SEND
  PROV -->|"no, interactive"| WAIT
  PROV -->|"no, deadline near"| DEG
  WAIT -->|"refilled"| SEND
  WAIT -->|"deadline passed"| NO

• Lanes and reserves. Interactive chat keeps a reserved share; batch work (ticket summaries, nightly evals) takes the rest or goes to a batch API at half price. Week 9, Day 2 · Noisy Neighbor Prevention, with tokens as the currency.
• Deadlines, not endless queues. A chat turn queued 10 seconds is useless: queue with a deadline, then degrade or refuse (Week 3, Day 3 · Backpressure and Flow Control).
• Retries with a budget. Retry 429 and 529 only before the first byte, with retry-after, jitter and a cap (Week 2, Day 1 · Timeout Hell); a provider that keeps failing trips a breaker (Week 2, Day 3 · Circuit Breakers) and Day 4's fallbacks take over.
• Idempotent sends. A double-tapped "send" gets one reply, keyed by a client message id (Week 2, Day 2 · Idempotency in Practice).
• Metering. Every usage, tagged with feature, tenant and model, goes onto a stream for Day 5's budgets and dashboards.

06Chapter 6: Cost per Token#

6.1 How Models Are Priced#

Anthropic and OpenAI both price input and output tokens separately, with discounts for cached input and batch work. List prices per million tokens, as of October 2026:

Sources: Anthropic docs, pricing; OpenAI docs, pricing. Both list batch processing at 50% off input and output.

The rows will change; the pattern matters: output costs 5-6× input, cached input 5-10% of input, and the table spans 50× on input price. So the calc below uses clearly labelled example prices.

6.2 The Worked Estimate#

Finance's Hindi question: if Hindi tokenises at, say, 3× the English count (an assumption; measure yours), output cost triples, so track cost per conversation by language. And on a model that thinks first, add thinking tokens to output.

07Chapter 7: How Real Companies Do It#

7.1 Uber: One Gateway for Every Model#

Uber found "disparate integration strategies adopted by different teams" and built the GenAI Gateway, "a unified platform for all LLM use cases within Uber" (Uber Engineering, July 2024). It mirrors "the HTTP/JSON interface of the OpenAI API" while reaching OpenAI, Google's Vertex AI and models Uber hosts itself, and handles authentication, metrics, audit logs for "cost attribution, security audit purposes, quality evaluation", and PII redaction before prompts reach vendors. It served "16 million queries per month, with a peak QPS of 25" for "close to 30 customer teams".

What carries over: Chapter 5's gateway. Note the peak QPS: its hard problems are tokens, cost and policy, not request rate.

7.2 LinkedIn: Latency Is a Product Decision#

Building its generative AI features, LinkedIn's team found that "TimeToFirstToken (TTFT) & TimeBetweenTokens (TBT) increase with utilization", that "it's not uncommon to get 2x/3x the TokensPerSecond (TPS)" by giving up some latency, and that chain-of-thought costs tokens "the member never sees". Their answer was to stream "within our pipeline end to end" and build "a fully async non-blocking pipeline that does not waste resources on account of threads blocked on I/O" (LinkedIn Engineering, April 2024).

What carries over: measure TTFT and TPOT under real load, budget invisible output tokens, and stream through every hop.

7.3 Character.AI: Self-Hosting at High Volume#

Character.AI calls itself "a full-stack AI company" that "designs its model architecture, inference stack and product from the ground up". It serves "around 20,000 queries per second" at "less than one cent per hour of conversation". It reports cutting serving costs "by at least 33X" since its 2022 launch, and says "It now costs us 13.5 times less to serve our traffic than it would cost a competitor building on top of the most efficient leading commercial APIs" (Character.AI, June 2024). It credits innovations around the Transformer architecture and the "attention KV cache", and "improved techniques for inter-turn caching", detailed in a separate technical post.

What carries over: at very high volume, owning the model and serving stack lets you attack costs an API customer can't touch, like KV cache size and caching between turns. At MegaMart's ~900 turns a minute, an API behind a gateway is the sensible start; revisit when busy GPU-hours would beat the per-token bill.

08Chapter 8: Common Mistakes#

1. Sizing by requests. "900 requests a minute is nothing." It's 3.45M input tokens a minute. Size, limit and price in tokens.
2. Re-sending everything forever. Full history, every policy, the whole order history, every turn. Set a context budget per feature; stable content first.
3. Treating 200 as success. Errors arrive after the 200, and max_tokens or a refusal can end a reply that looks complete. Store every reply's stop reason.
4. A buffering hop. One buffering proxy turns streaming back into a spinner. Test end to end through production's real proxies.
5. Retrying blindly. Immediate 429 retries feed the storm, retries after the first byte duplicate output, and a spend-cap 429 never succeeds. Retry before the first byte, with retry-after, jitter and a budget.

09Chapter 9: Interview Tips#

9.1 When to Bring It Up#

Whenever a design says "and then we call the LLM": a support bot, a summariser, generated search answers, a coding assistant. One sentence shows you see the distributed-systems problem: "the model call is a slow, metered, token-limited dependency, so I'll put a gateway in front of it and budget tokens."

9.3 Questions to Ask the Interviewer#

• "A vendor API or a model we host? Any data-residency rules?"
• "Which latency matters: first words or the whole answer? Read by a person or parsed by code?"
• "How long are conversations, what must be in context, and which languages do customers use?"

9.4 Common Follow-Up Questions#

10Chapter 10: Practice Problems#

Summary#

Further Reading#

Papers#

• Paper

  Yu et al. — Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022): https://www.usenix.org/conference/osdi22/presentation/yu

• Paper

  Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/abs/2309.06180

• Paper

  Dao et al. — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022): https://arxiv.org/abs/2205.14135

• Paper

  Petrov et al. — Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023): https://arxiv.org/abs/2305.15425

Engineering Posts#

• Blog

  Databricks — LLM Inference Performance Engineering: Best Practices (12 October 2023): https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices

• Blog

  Anyscale — How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (22 June 2023): https://www.anyscale.com/blog/continuous-batching-llm-inference

• Blog

  LinkedIn — Musings on building a Generative AI product (25 April 2024): https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product

• Blog

  Uber — Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway (11 July 2024): https://www.uber.com/blog/genai-gateway/

• Blog

  Character.AI — Optimizing AI Inference at Character.AI (20 June 2024): https://blog.character.ai/optimizing-ai-inference-at-character-ai

Documentation (read October 2026)#

• Docs

  Anthropic — Pricing: https://platform.claude.com/docs/en/about-claude/pricing

• Docs

  Anthropic — Rate limits: https://platform.claude.com/docs/en/api/rate-limits

• Docs

  Anthropic — Errors: https://platform.claude.com/docs/en/api/errors

• Docs

  Anthropic — Streaming messages: https://platform.claude.com/docs/en/build-with-claude/streaming

• Docs

  Anthropic — Context windows: https://platform.claude.com/docs/en/build-with-claude/context-windows

• Docs

  Anthropic — Token counting: https://platform.claude.com/docs/en/build-with-claude/token-counting

• Docs

  Anthropic — Effort: https://platform.claude.com/docs/en/build-with-claude/effort

• Docs

  OpenAI — Latency optimization: https://developers.openai.com/api/docs/guides/latency-optimization

• Docs

  OpenAI — Rate limits: https://developers.openai.com/api/docs/guides/rate-limits

• Docs

  OpenAI — Pricing: https://developers.openai.com/api/docs/pricing

• Docs

  Google — Gemini API: Understand and count tokens: https://ai.google.dev/gemini-api/docs/tokens

• Docs

  NVIDIA — NIM LLM benchmarking metrics: https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html

• Docs

  vLLM — Online serving (OpenAI-compatible server): https://docs.vllm.ai/en/latest/serving/online_serving/

• Docs

  nginx — ngx_http_proxy_module (proxy_buffering, X-Accel-Buffering, proxy_read_timeout): https://nginx.org/en/docs/http/ngx_http_proxy_module.html

In This Course#

• Read

  Side quest 14 — ChatGPT LLM Serving (the provider's side: KV sizing, schedulers, fleet routing): /learn/system-design-mastery/bonus-problems/bonus-14-chatgpt-llm-serving