~learn.

Side quest 14LLM serving

ChatGPT LLM Serving.

Hundreds of Millions of People, One Token at a Time, and a GPU Bill Measured in Megawatts

EXPERT40 MIN READ6 PHASES · 45 MIN INTERVIEW+120 XP

Mission brief

Serve a large language model to hundreds of millions of people

Quest log0/7 stages · 0/8 checkpoints · 0/8 quick checks
ChatGPT weekly users (OpenAI DevDay, September 2026)
1.2 billion
prompts a day (OpenAI, July 2025)
2.5 billion
to the first token (design target)
~1 s
per step, for every reply in flight
One token

Company names and logos are trademarks of their owners, shown only to identify each company. This course isn't affiliated with or endorsed by any of them.

Optional · timed run · 6 phases · 45 min

Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.

Intro · 3 min

How Do You Stream a Reply One Token at a Time to Hundreds of Millions of People Without Melting the GPUs?

It is 8:40 pm in Bengaluru. Meera types "Plan a three-day trip to Hampi with a toddler" and presses enter. Within a second the first words appear, and the rest of the reply flows in faster than she can read it.

In that second her message was checked, joined to the earlier turns of her conversation and squeezed onto a GPU alongside a couple of hundred other people's replies. Every few tens of milliseconds the GPU writes one more token for each of them; her reply is a few hundred such steps.

More servers can't make one reply faster, because the model writes one token after another. The design has to make each GPU serve as many people at once as it can, without anyone noticing they are sharing.

ChatGPT serving · a prompt joins a queue, is admitted into a free slot of a GPU's running batch, and tokens stream out one step at a time while finished replies free their slots for the next prompts
LIVEChatGPT serving · a prompt joins a queue, is admitted into a free slot of a GPU's running batch, and tokens stream out one step at a time while finished replies free their slots for the next prompts
ChatGPT, from published numbers
  • 1.2 billion — weekly users in September 2026, shown at OpenAI's DevDay; up from 900 million in February 2026, when OpenAI reported 50 million paying subscribers
  • 2.5 billion — prompts a day in July 2025, about 330 million of them from the US (OpenAI, via Axios)
  • 6 billion+ — tokens a minute through OpenAI's API in October 2025 (Sam Altman, DevDay)
  • 1 million — new users in one hour after image generation reached everyone in March 2025; the original launch took five days to reach 1 million (Sam Altman)
  • "Our GPUs are melting" — Sam Altman, March 2025, announcing temporary rate limits on image generation

For context:
A token is a piece of a word: OpenAI's rule of thumb is 1 token ≈ 4 characters ≈ ¾ of an English word
OpenAI has published little about how ChatGPT is served; everything not sourced here is how you would design it, not OpenAI's internals
The sizing uses an open 70-billion-parameter model of known shape, not any OpenAI model

This is the system we'll design today.

ExplainerThe whole quest: ChatGPT serving

ChatGPT serves 1.2 billion weekly users: ~100,000 messages a second at peak is 50 million output tokens a second, and a 4-GPU replica fits ~225 conversations because KV cache, not weights, fills memory. Messages pass the edge, context assembly and a router to a GPU replica, and tokens stream back through a resumable token log. The deep dives cover memory-bound decode, continuous batching, paged KV and prefix caching, tier lanes and slow cold starts, then a dead replica, the degradation ladder and a 5x launch day.

The Interview Begins#

You're interviewing at a company that runs a popular AI assistant. The interviewer opens a graph of GPU utilisation across the fleet: a wave that follows the sun, with a sharp spike on the day a new feature launched.

Interviewer

"We have the model; training is someone else's problem. Your job is serving it: a message in, a reply streaming back, for over a billion people a week, without the GPU bill or the latency getting out of hand."

Serve a large language model to hundreds of millions of people

Users send messages in conversations from web, mobile and desktop apps. Each reply is generated by a large model on GPUs and streamed back as it is written. Free and paid users share the fleet; paid users get more and faster.

Requirements
  • Show the first words of a reply within about a second, and keep them coming faster than anyone reads
  • Use the GPUs well: they are the cost of the whole business
  • Be fair: paid tiers keep their promises under load, free users get an honest answer, not a spinner
  • Check inputs and outputs for safety without making every reply wait
  • Survive a GPU failing mid-reply, a region going down and a launch at 5× demand
Interviewer

"Assume the weights exist. Everything from the keyboard to the GPU and back is yours."

Phase 1 · Stage 1 of 7 · 5 min budget · 2 min read

Phase 1: Requirements Clarification (5 minutes)#

You

"I want to pin down what 'fast' and 'cheap' mean for LLM serving."

Your Questions#

Transcript
You

"How many users and messages?"

Interviewer

"ChatGPT showed 1.2 billion weekly users at OpenAI's DevDay in September 2026, and reported 2.5 billion prompts a day in July 2025, when it was approaching 700 million weekly users. Design for 1.2 billion weekly users."

You

"What does fast mean for a reply that takes twenty seconds to write?"

Interviewer

"Two numbers. Time to first token: how long until the first word appears. Then time per output token: how fast the rest arrives. People forgive a long reply; they don't forgive a blank screen."

You

"Are there tiers?"

Interviewer

"Free, paid and business plans. OpenAI has said that once usage limits are reached, a mini version of each model handles the remaining queries, and describes GPT-5 as a fast model and a reasoning model behind a real-time router."

You

"Where does safety sit?"

Interviewer

"In the path: inputs before generation, outputs while they stream. OpenAI's API documentation warns that streaming makes moderation harder, because partial completions are harder to judge."

Functional Requirements#

Functional requirements
  • Chat: send a message and get a streamed reply; stop it; regenerate
  • Conversations: history on every device; each reply sees the earlier turns
  • Models and tiers: several models (fast, reasoning, mini); per-tier limits; fallback to a smaller model past a limit
  • Safety: input checked before generation, output during streaming; a reply can be stopped and replaced
  • Accounting: tokens per user and per model, for limits, billing and cost

Non-Functional Requirements#

Non-functional requirements
Latency (design targets, at our edge, excluding visible "thinking" time)
  • Time to first token (TTFT): p95 < 1 s for paid tiers, < 2 s for free
  • Time per output token (TPOT): p95 < 50 ms, i.e. at least 20 tokens a second per reply
  • No stalls: 99.9% of streams never pause for more than 2 s mid-reply
Availability and fairness
  • 99.9% of messages get a complete reply, or an honest, actionable error, per 30 days
  • Under overload, paid tiers keep their latency targets; free users get a clear limit, a smaller model or an honest wait
Cost
  • GPU cost per million output tokens and per message are first-class metrics; the fleet is sized for the peak, so utilisation is money
You

"Why 20 tokens a second? Say people read 250 words a minute: at ¾ of a word per token, 250 ÷ 60 ÷ 0.75 ≈ 5.6 tokens a second. Twenty leaves room to skim; much faster mostly buys nothing."

Phase 2 · Stage 2 of 7 · 5 min budget · 4 min read

Phase 2: Back of the Envelope Estimation (5 minutes)#

You

"Requests are the wrong unit. A GPU's work is tokens read in (prefill) and tokens written out (decode). Tokens a second first, then GPUs, memory and power."

Your turn · checkpoint 1not answered

Before the numbers: how many tokens a second?

Design for 4.3 billion messages a day. If an average reply is 500 tokens and the evening peak is twice the daily average, how many output tokens a second does the fleet write at peak?

Think it through, or jot it down. Then compare.

One strong answer

  • Messages: 4.3 billion ÷ 86,400 s ≈ 49,800 a second on average, call it 50,000; at 2× peak, ~100,000 a second.
  • Output tokens: 50,000 × 500 = 25 million tokens a second on average; ~50 million a second at peak.
  • Input tokens: if each message carries ~2,000 tokens of context (instructions, history, the new message), that's 100 million tokens a second read on average.

Each written token is one step of the model through all its weights.

Traffic Estimation#

WorksheetMessages and tokens (design targets; assumptions labelled)
Average Evening peak (2×, assumption)
Messages a day (design: 1.2 billion weekly users, scaled from 2.5 billion at ~700 million) 4.3 billion —
Messages a second ~50,000 ~100,000
Output tokens a second (500 a reply, assumption, including any hidden reasoning) 25 million 50 million
Input tokens a second (2,000 a message, assumption) 100 million 200 million
Input tokens computed, if 50% are served from prefix caches (assumption) 50 million 100 million
  • Conversations a day at ~4 messages each (assumption): 4.3 billion ÷ 4 ≈ 1.1 billion
  • Sanity check: 100 + 25 = 125 million tokens a second; OpenAI's API alone reported 6 billion a minute (100 million a second) in October 2025, the same order of magnitude

GPU Memory: Weights vs KV Cache#

You

"OpenAI doesn't publish its models' sizes, so I'll size with an open model, Llama 3.3 70B: 80 layers, 64 query heads but only 8 key-value heads (grouped-query attention), head dimension 128, 128K-token context."

WorksheetOne 70B model on H100 GPUs (80 GB, 3.35 TB/s each; NVIDIA)
  • Weights: 70 billion × 2 bytes (bf16) = 140 GB; in fp8, 70 GB. Neither leaves room on one 80 GB GPU, so the model is split across GPUs (tensor parallelism)
  • KV cache per token = 2 (K and V) × layers × kv_heads × head_dim × bytes = 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.33 MB in bf16 (0.16 MB in fp8)
  • Per sequence: a 2,000-token conversation = 2,000 × 327,680 ≈ 655 MB; a full 128K-token context (131,072 tokens) ≈ 43 GB, more than half a GPU for one user
  • Without grouped-query attention (64 KV heads, not 8) the cache would be 8× bigger: 2.6 MB a token
Replica (illustrative) 4 × H100, bf16 2 × H100, fp8 weights + fp8 KV
GPU memory 320 GB 160 GB
Weights 140 GB 70 GB
Reserved for activations, runtime (~10%) 32 GB 16 GB
Left for KV cache 148 GB 74 GB
Tokens of KV cache 148 GB ÷ 327,680 B ≈ 450,000 74 GB ÷ 163,840 B ≈ 450,000
Sequences of 2,000 tokens ~225 (~56 a GPU) ~225 (~112 a GPU)

Conclusion: after the weights, KV cache memory decides how many people a replica serves at once; fp8 serves the same 225 on half the GPUs, if quality holds.

Throughput, GPUs and Power#

WorksheetWhy ~1,500 output tokens a second per GPU (illustrative)
  • One decode step reads all the weights once, plus every sequence's KV cache: 140 GB + 225 × 655 MB ≈ 288 GB per step on the 4-GPU replica
  • At 4 × 3.35 TB/s = 13.4 TB/s: 288 GB ÷ 13.4 TB/s ≈ 21.5 ms a step at best → ~46 tokens a second per stream, ~10,500 tokens a second per replica
  • At a realistic ~60% of peak bandwidth (assumption): ~36 ms a step, ~28 tokens a second per stream, ~1,500 tokens a second per GPU
  • Prefill is compute: 2 FLOPs per parameter per token = 1.4 × 10¹¹ FLOPs a token. H100 dense bf16 is ~989 TFLOPS; at 50% (assumption) ≈ 3,500 prompt tokens a second per GPU. A 2,000-token prompt on 4 GPUs: ~140 ms
WorksheetThe fleet, if every message ran on this one model (it wouldn't; real fleets mix models)
  • Decode at peak: 50 million ÷ 1,500 ≈ 33,300 GPUs
  • Prefill at peak: 100 million uncached tokens ÷ 3,500 ≈ 28,600 GPUs
  • Together ≈ 62,000; with ~20% headroom for failures and spikes, ~75,000 H100-class GPUs
  • Cross-check with Little's law: a reply of 500 tokens at ~28 a second takes ~18 s; 100,000 a second × 18 s ≈ 1.8 million streams in flight. At 225 per 4-GPU replica that's ~8,000 replicas ≈ 32,000 GPUs for decode, close to the 33,300 above
  • Power: 75,000 GPUs ≈ 9,400 eight-GPU servers × 10.2 kW (DGX H100 maximum) ≈ 96 MW; at a PUE of 1.2 (assumption) ≈ 115 MW, a small power station
  • Cost: at $2.50 per GPU-hour (assumption), 75,000 × $2.50 × 24 = $4.5 million a day, ≈ $0.001 a message; per million output tokens, a GPU at 1,500 tokens a second writes 5.4 million an hour, ≈ $0.46 at full use, but the fleet averages ~40% busy (31,000 GPUs of work against 75,000), so all-in it is nearer $2

Key Metrics Summary#

Metric Value
Messages a second, average / peak (design) ~50,000 / ~100,000
Output tokens a second, average / peak 25 million / 50 million
Sequences per 4 × H100 replica ~225
Streams in flight at peak ~1.8 million
GPUs (illustrative, one 70B model for everything) ~75,000 H100-class
Power ~115 MW
Quick check3 questions · +5 XP each

Requirements and estimation

  1. Q1/3 A 70B model has 80 layers, 8 key-value heads and a head dimension of 128. How big is its KV cache per token in bf16?

    Answer & explanation

    Answer: B

    2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes. With 64 KV heads instead of 8 it would be 2.6 MB.

  2. Q2/3 Why is time to first token measured separately from time per output token?

    Answer & explanation

    Answer: B

    TTFT covers admission, queueing and reading the whole prompt; TPOT is the steady pace of decode steps. They fail for different reasons.

  3. Q3/3 After the weights are loaded, what mostly limits how many replies one replica can generate at once?

    Answer & explanation

    Answer: C

    Every sequence in the batch keeps its keys and values in GPU memory; 148 GB holds ~450,000 tokens, about 225 conversations of 2,000 tokens.

Phase 3 · Stage 3 of 7 · 10 min budget · 3 min read

Phase 3: High-Level Design (10 minutes)#

Your turn · checkpoint 2not answered

Sketch it first: one message, keyboard to screen

Trace Meera's message from the app to the GPU and back. Which parts are ordinary web services, which part is scarce and expensive, and where does the reply live while it is being written?

Think it through, or jot it down. Then compare.

One strong answer

  • Ordinary and cheap: edge, auth, quotas, conversations, storage: any web tier.
  • Scarce and expensive: GPU replicas. Everything in front of them keeps them full of useful work and away from work they can't finish in time.
  • The router and schedulers pick the model, region and replica for each request, and the order.
  • While it streams, the reply lives twice: in the replica's KV cache (to write the next token) and in a short-lived token log (to resend tokens after a reconnect). It reaches the conversation store when it ends.
  • Safety is in the path: input before generation, output while it streams.
You

"The GPU tier is the only part that is hard to scale and expensive to waste. Everything else is designed around keeping it busy and fair."

System Architecture#

DiagramLLM serving architecture
%% title: LLM serving architecture
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    APP["ChatGPT apps<br/>web, mobile, desktop"]:::client
    DEV["API clients<br/>another front door"]:::client
  end
  subgraph L2["Layer 2 · Edge"]
    direction LR
    EDGE["Edge + TLS<br/>long-lived SSE streams"]:::edge
    GW["API gateway<br/>auth, quotas, limits"]:::edge
  end
  subgraph L3["Layer 3 · Conversation"]
    direction LR
    CONV["Conversation service<br/>context assembly"]:::service
    SAFE["Safety<br/>input + output checks"]:::service
    HIST[("Conversation store<br/>replicated")]:::store
  end
  subgraph L4["Layer 4 · Scheduling"]
    direction LR
    ROUTE["Router<br/>model, region, prefix"]:::core
    LANES[["Tier lanes<br/>paid, free, background"]]:::queue
  end
  subgraph L5["Layer 5 · Inference"]
    direction LR
    REP["Model replicas<br/>4 GPUs, batch + KV cache"]:::hot
    MINI["Small-model pool<br/>fallback, routing"]:::service
    TLOG[("Token log<br/>replies in flight")]:::cache
  end
  subgraph L6["Layer 6 · Operations"]
    direction LR
    TEL["Telemetry<br/>TTFT, TPOT, cost"]:::service
    FLEET["Fleet manager<br/>capacity, spot, drains"]:::service
    BILL[["Usage events<br/>limits, billing"]]:::queue
  end
  L1 ==> L2
  L2 ==>|"message"| L3
  L3 ==>|"generate"| L4
  L4 ==>|"admit to a replica"| L5
  L5 -->|"tokens, usage, health"| L6

KEY INSIGHTS:

  • The router and the replicas' schedulers are the core; the rest is a well-known web tier
  • Replies stream back up the same path (relay, conversation service, edge) over long-lived connections: ~1.8 million open at peak
  • Tokens pass through a relay with a short log, so a dropped connection doesn't lose work the GPU already paid for

Data Flow: One Message at 20:40#

DiagramOne message, end to end
%% title: One message, end to end
sequenceDiagram
  autonumber
  participant A as Meera's app
  participant E as Edge + gateway
  participant C as Conversation svc
  participant S as Scheduler
  participant M as Replica r17
  Note over A: 20:40:03.000 · "Plan a 3-day trip to Hampi"
  A->>E: Send message (conversation c42, key m9)
  E->>E: Auth · quota ok (paid tier)
  E->>C: Forward · open the stream back
  C->>C: Input check · build 2,000-token context
  C->>S: Generate (model, tier, prefix hash)
  S->>M: Admit · prefix already cached here
  M->>M: Prefill the ~300 new tokens
  M-)C: Token 1 at 20:40:03.420
  C-)A: First words on screen (through the edge)
  loop every ~35 ms until a stop token
    M-)C: Next token
    C-)A: Stream event, numbered
  end
  M->>M: Free the KV blocks
  C->>C: Output check · save the reply
  C-)A: Done

MEANWHILE, IN PARALLEL:

  • Replica r17 is writing ~224 other replies; each decode step adds a token to all of them
  • The output check runs on each finished sentence, alongside the stream
Quick check3 questions · +5 XP each

High-level design

  1. Q1/3 Which tier of the design is scarce and expensive enough that everything else is built around it?

    Answer & explanation

    Answer: C

    The rest are ordinary web services. Every design choice in front of the GPUs is about keeping them full of useful work.

  2. Q2/3 Why do replies pass through a relay with a short token log?

    Answer & explanation

    Answer: B

    The GPU already paid for those tokens. A log lets a reconnecting client ask for "everything after token 212".

  3. Q3/3 A reply takes ~18 s and 100,000 replies start every second at peak. Roughly how many streams are open?

    Answer & explanation

    Answer: C

    Little's law: in flight = rate × time = 100,000 × 18 s ≈ 1.8 million.

Phase 4 · Stage 4 of 7 · 20 min budget · 17 min read

Phase 4: Deep Dives (20 minutes)#

Deep Dive 1: Why LLM Serving Is Different (Week 0 Concepts: Back of the Envelope, Operating Systems)#

You

"A web request does a fixed amount of work. A model's reply is a loop: one step per token, each depending on the one before. That changes where time and memory go."

Your turn · checkpoint 3not answered

One user on a whole replica

A 4-GPU replica serves exactly one user. The 70B model is in bf16 and the conversation is 2,000 tokens. How fast does that user get tokens, and how busy are the GPUs' compute units? Now put 225 users on it. What changes?

Think it through, or jot it down. Then compare.

One strong answer

  • One user: each step reads 140 GB of weights + 0.66 GB of KV cache at 13.4 TB/s ≈ 10.5 ms: ~95 tokens a second. 1.4 × 10¹¹ FLOPs in 10.5 ms is ~13 TFLOPS of ~3,956 available: 0.3% busy.
  • The reason: decode does about 1 FLOP per byte read (2 FLOPs per 2-byte weight), while an H100 can do ~295 FLOPs per byte its memory delivers. Decode at batch 1 is memory-bound.
  • 225 users: the weights are read once per step for everyone. The step grows to ~21.5 ms (each sequence's KV adds reads), so each user gets ~46 tokens a second, but the replica writes ~10,500 a second, 110× more, and compute is ~37% busy.
  • Batching is the whole economics: each user loses a little speed; the GPU's output multiplies.

The Challenge:

What makes an LLM unlike a web service
  • Work isn't known in advance: a reply may stop after 5 tokens or 5,000, so neither its duration nor its memory is known when it starts
  • Two phases with opposite needs: reading the prompt is compute-heavy; writing the reply is memory-bandwidth-heavy
  • State grows while it runs: every token adds 0.33 MB to the sequence's KV cache, on the GPU, until the reply ends
  • Steps are sequential: token 101 can't start before token 100, so a slow step slows every reply in the batch

The Solution:

DiagramPrefill and decode
%% title: Prefill and decode
flowchart TD
  P(["Prompt · ~2,000 tokens"]):::client
  subgraph PF["Prefill · once per request"]
    PF1["All prompt tokens at once<br/>large matrix multiplies"]:::core
    PF2["Compute-bound<br/>~140 ms on 4 GPUs"]:::service
  end
  KV[("KV cache<br/>0.33 MB per token")]:::cache
  subgraph DC["Decode · once per output token"]
    D1["One token for each sequence<br/>reads all weights + its KV"]:::hot
    D2{"Stop token or<br/>max tokens?"}:::service
  end
  OUT(["Token streamed to the user"]):::ok
  DONE(["Finished · KV blocks freed"]):::ok
  P --> PF1 --> PF2 -->|"writes K, V for every prompt token"| KV
  KV --> D1 --> OUT
  D1 --> D2
  D2 -->|"no · append this token's K, V"| KV
  D2 -->|"yes"| DONE
  • Prefill reads the whole prompt in parallel and writes its keys and values into the KV cache. It sets the time to first token.
  • Decode produces one token per step for every sequence in the batch, reading all the weights and each sequence's cache. It sets the time per output token.
  • The KV cache is why decode doesn't recompute the prompt every step, and why memory runs out.
You

"Two optimisations follow. fp8 weights and KV cache halve the bytes per step and per token. And since the phases want different things, they can be separated: DistServe (OSDI 2024) runs prefill and decode on different GPUs, while Sarathi-Serve (OSDI 2024) cuts long prompts into chunks that ride along with decode steps."

Deep Dive 2: Batching, Paged KV Memory and Prefix Caching (Week 4 Concepts: Caching, Week 0: Operating Systems)#

You

"So batching is the economics. The question is how to batch requests that start at different times, stop at unknown times and grow as they go."

Your turn · checkpoint 4not answered

A batch of eight, one long reply

A replica batches 8 requests and runs them until all finish. Seven replies are 50 tokens long; one is 2,000. The KV cache is reserved per request for the 4,096-token maximum. What does that cost, and how would you fix it?

Think it through, or jot it down. Then compare.

One strong answer

  • Static batching wastes time: the 7 short replies finish at step 50, but their slots sit empty until the long one finishes at step 2,000. For 97.5% of the batch's life it runs 1 of 8.
  • New requests wait for the whole batch, so their time to first token includes someone else's long reply.
  • Reserving for the maximum wastes memory: 4,096 tokens × 0.33 MB ≈ 1.3 GB held per request, when most use a fraction.
  • Fix 1, continuous batching: schedule per step, not per batch. A finished sequence leaves after its step and a waiting one joins at the next. This is Orca's iteration-level scheduling (OSDI 2022).
  • Fix 2, paged KV memory: allocate the cache in small blocks (16 tokens) on demand, like virtual-memory pages, so a sequence holds only what it has used. This is vLLM's PagedAttention (SOSP 2023).

The Solution:

What the papers measured
  • 36.9× — Orca's throughput over NVIDIA FasterTransformer at the same latency, on GPT-3 175B (OSDI 2022)
  • 20.4%–38.2% — share of KV cache memory holding actual tokens in the systems vLLM's authors measured; the rest lost to reservation and fragmentation
  • 2–4× — vLLM's throughput over FasterTransformer and Orca at the same latency (SOSP 2023)
  • Up to 6.4× — SGLang's throughput, helped by RadixAttention reusing KV cache across shared prefixes
ExplainerLLM batching: static, continuous, paged KV

In a static batch of 8, seven 50-token replies finish at step 50 and their slots idle until the 2,000-token reply ends: 1 of 8 runs for 97.5% of the batch, and new requests wait. Continuous batching schedules per step, so finished sequences leave and waiting ones join at the next step. Paged KV memory replaces a 4,096-token reservation (~1.3 GB a request) with 16-token blocks on demand, and prefix caching shares full blocks of the same instructions.

DiagramA request inside one replica
%% title: A request inside one replica
stateDiagram-v2
  [*] --> WAITING: routed to this replica
  WAITING --> RUNNING: blocks free for its prompt
  RUNNING --> PREEMPTED: out of blocks, newest evicted
  PREEMPTED --> WAITING: requeued, prefix recomputed
  RUNNING --> FINISHED: stop token or max tokens
  RUNNING --> CANCELLED: user stopped or left
  WAITING --> CANCELLED: user left before it started
  FINISHED --> [*]
  CANCELLED --> [*]
  WAITING: WAITING · in a tier lane
  RUNNING: RUNNING · one token per step
  PREEMPTED: PREEMPTED · blocks taken back
  class RUNNING core
  class WAITING,PREEMPTED queue
  class FINISHED ok
  class CANCELLED bad

Prefix caching is the third lever. Many prompts share a beginning:

  • Shared instructions: every conversation starts with the same system instructions and tool descriptions
  • The conversation so far: turn 6 of Meera's chat repeats turns 1 to 5
  • Keep those blocks: hash each full block with everything before it; a request whose prefix hashes match shares those blocks (reference-counted) and prefills only the new part
  • Published behaviour: OpenAI's API documentation describes cached prefixes lasting minutes (typically 5 to 10 of inactivity on earlier models, 30 after the last use on its newest) and discounts cached input tokens
pythonBlockPool · 98 lines
# engine/scheduler.py

"""
Continuous batching with paged KV memory and a prefix cache (illustrative,
in the style of vLLM). One call = one step for the whole running batch.
Only FULL prompt blocks are shared, so nobody writes into a shared block;
a freed block keeps its cached prefix until it is handed out again.
"""

from collections import OrderedDict, deque

BLOCK = 16                                        # tokens per block


class BlockPool:
    def __init__(self, n_blocks: int):
        self.free = OrderedDict.fromkeys(range(n_blocks))   # least recently freed first
        self.refs, self.by_hash, self.hash_of = {}, {}, {}

    def take(self, h=None) -> int:
        b, _ = self.free.popitem(last=False)      # evicts the oldest cached prefix
        old = self.hash_of.pop(b, None)
        if old is not None:
            del self.by_hash[old]
        if h is not None:
            self.by_hash[h], self.hash_of[b] = b, h
        self.refs[b] = 1
        return b

    def reuse(self, h):
        b = self.by_hash.get(h)
        if b is not None:
            if self.refs.get(b, 0) == 0:
                del self.free[b]                  # revive a cached, freed block
            self.refs[b] = self.refs.get(b, 0) + 1
        return b

    def release(self, b: int) -> None:
        self.refs[b] -= 1
        if self.refs[b] == 0:
            self.free[b] = None


class Scheduler:
    def __init__(self, pool: BlockPool, max_batch: int = 256, reserve: int = 64):
        self.pool, self.max_batch, self.reserve = pool, max_batch, reserve
        self.waiting, self.running = deque(), []  # waiting: filled by the router, by lane

    def _blocks_for(self, seq):
        full = len(seq.prompt) // BLOCK
        blocks = []
        for i in range(-(-seq.length // BLOCK)):  # a preempted seq re-prefills its output too
            h = seq.prefix_hash(i) if i < full else None
            b = self.pool.reuse(h) if h is not None else None
            if b is not None:
                seq.cached_tokens += BLOCK        # no prefill for this block
            elif len(self.pool.free) > self.reserve:
                b = self.pool.take(h)             # keep a reserve for running seqs to grow
            else:
                for b2 in blocks:
                    self.pool.release(b2)
                seq.cached_tokens = 0
                return None
            blocks.append(b)
        return blocks

    def step(self, model) -> None:
        for seq in [s for s in self.running if s.done or s.cancelled]:
            self._free(seq)                       # 1. finished and cancelled leave first
        while self.waiting and len(self.running) < self.max_batch:
            blocks = self._blocks_for(self.waiting[0])
            if blocks is None:
                break                             # 2. admit until memory is tight
            seq = self.waiting.popleft()
            seq.blocks = blocks
            self.running.append(seq)
        for seq in list(self.running):            # 3. room for each sequence's next token
            if seq not in self.running or seq.length < len(seq.blocks) * BLOCK:
                continue
            while not self.pool.free and len(self.running) > 1:
                self._preempt(max(self.running, key=lambda s: s.arrived))   # newest first
            if seq in self.running:
                if self.pool.free:
                    seq.blocks.append(self.pool.take())
                else:
                    self._preempt(seq)
        model.forward(self.running)               # 4. prefill arrivals, one token for the rest

    def _free(self, seq) -> None:
        for b in seq.blocks:
            self.pool.release(b)
        seq.blocks = []
        self.running.remove(seq)

    def _preempt(self, victim) -> None:
        self._free(victim)
        victim.cached_tokens = 0
        self.waiting.appendleft(victim)           # recomputed when blocks free up

A few details carry the design:

  • Stop admitting before memory runs out: preemption throws work away, so a reserve of blocks is kept for running sequences to grow into.
  • Preempt the newest: it has the least to redo. vLLM's paper evicts whole sequences and swaps their blocks to CPU memory or recomputes them later.
  • A cancelled sequence frees its blocks at the next step, so "Stop" saves money, not just attention.
  • Real engines also cache full blocks written during decode, so turn 5's reply is cached for turn 6; the sketch hashes prompt blocks only.
Quick check4 questions · +5 XP each

Prefill, decode and batching

  1. Q1/4 Why does a replica's throughput rise ~110× when it goes from 1 to 225 sequences?

    Answer & explanation

    Answer: C

    Decode is memory-bound. One read of 140 GB of weights serves every sequence in the batch, so each extra sequence costs only its own KV reads.

  2. Q2/4 In static batching, seven 50-token replies share a batch with one 2,000-token reply. What happens to the seven slots after step 50?

    Answer & explanation

    Answer: A

    That idle time is what continuous (iteration-level) batching removes: slots are refilled every step.

  3. Q3/4 What does PagedAttention change about KV cache memory?

    Answer & explanation

    Answer: C

    Blocks of 16 tokens, mapped like virtual-memory pages, cut waste to under one block per sequence and let shared prefixes share blocks.

  4. Q4/4 Turn 6 of a conversation arrives at the replica that served turn 5 two minutes ago. What can prefix caching skip?

    Answer & explanation

    Answer: B

    The earlier turns' KV blocks are still cached, so only the new message has to be prefilled. Arriving at a different replica would lose this.

Deep Dive 3: Scheduling and Routing Across the Fleet (Week 1 Concepts: Rate Limiting, Week 9: Noisy Neighbours, Week 10: Capacity Planning)#

You

"Inside a replica, the scheduler fills a batch. Across the fleet, the router picks the replica and the fleet manager decides how many there are, both respecting tiers and warm caches."

Your turn · checkpoint 5not answered

The evening peak and the free tier

It's 21:00 in India and 16:30 in Europe. One region's replicas are 95% full. Free users send about 85% of the messages. A paid user's message and three free users' messages arrive in the same millisecond. Who goes first, where does each go, and what does a free user see if the region stays full for twenty minutes?

Think it through, or jot it down. Then compare.

One strong answer

  • Lanes by tier: paid interactive, then free interactive, then background (batch jobs, evaluations, flex-tier API traffic). Within a lane, round-robin by user so one heavy user can't fill it.
  • Admission by memory: admit only if the prompt's blocks fit and the TTFT estimate is inside the lane's target.
  • Route for the cache: to the replica holding the conversation's prefix, unless its queue is over a threshold; then the least-loaded one, paying the prefill.
  • Spill across regions: a neighbouring region costs ~100 ms of network but saves seconds of queueing.
  • Twenty minutes full: free users past their limit get the smaller model (as ChatGPT does); if that is full too, "high demand, try again in about 10 minutes", not a spinner.

The Challenge:

Ways a fleet of model replicas goes wrong
  • Round-robin routing scatters a conversation's turns across replicas, so every turn re-prefills its whole history
  • Pure cache affinity turns one popular prefix into one hot replica
  • One queue for everyone lets a free-tier surge push paid TTFT past its target
  • Reactive autoscaling asks for GPUs when the queue is long; a new replica takes minutes to be useful

The Solution:

DiagramRouting one request
%% title: Routing one request
flowchart TD
  REQ(["Generate request"]):::client
  TIER{"Within the tier's<br/>limits?"}:::service
  FALL(["Mini model<br/>or a clear limit message"]):::bad
  MODEL["Choose the model<br/>fast, reasoning or mini"]:::core
  REGION["Choose the region<br/>nearest with headroom"]:::edge
  AFF{"A replica holds<br/>this prefix?"}:::cache
  LOAD{"Its queue under<br/>the threshold?"}:::service
  HOT(["Send there<br/>skip most prefill"]):::ok
  LEAST(["Least-loaded replica<br/>prefill from scratch"]):::service
  subgraph Q["Lanes on every replica"]
    direction LR
    Q1[["1 · Paid interactive"]]:::queue
    Q2[["2 · Free interactive"]]:::queue
    Q3[["3 · Background, flex"]]:::queue
  end
  REQ --> TIER
  TIER -->|"over"| FALL
  TIER -->|"within"| MODEL --> REGION --> AFF
  AFF -->|"yes"| LOAD
  AFF -->|"no"| LEAST
  LOAD -->|"yes"| HOT
  LOAD -->|"no · overflow"| LEAST
  HOT --> Q
  LEAST --> Q
  • Quotas in tokens, not requests: a token bucket per user per model, charged with the prompt plus the requested maximum output. OpenAI's API limits requests and tokens per minute (RPM, TPM), counts the larger of max_tokens and an estimate of the request against TPM, and returns 429 with rate-limit headers.
  • Prefix affinity with overflow, the published shape of OpenAI's prompt-cache routing: a hash of the first tokens picks the machine; heavy prefixes spill to more machines.
  • Lanes, not one queue: paid work is admitted first; background work fills the gaps, so idle GPU-seconds become cheap batch output.

Autoscaling GPUs That Take Minutes to Start#

WorksheetCold start of one 4-GPU replica (illustrative)
  • Get a machine: seconds from a warm pool, minutes from a cloud provider, never if it is out of capacity
  • Pull the serving image: ~1 minute
  • Load 140 GB of weights: at ~2 GB/s from network storage ≈ 70 s; from local NVMe at ~10 GB/s ≈ 14 s
  • Warm up (compile kernels, capture CUDA graphs, health checks): ~1 minute
  • Total: ~3 to 10 minutes before it serves its first token, and its prefix cache starts cold
You

"So the fleet scales on a forecast, not a queue. The daily wave follows the sun and repeats, so capacity is placed ahead of it; reactive scaling covers forecast error, and the degradation ladder covers the minutes in between."

Where the GPUs come from

Reserved or owned

  • The baseline, busy every hour of the day
  • Cheapest per hour when used; paid for when not

On-demand and spot

  • The evening peak and launch days
  • Spot is cheaper but can be taken back: AWS gives a two-minute interruption notice
  • On notice: stop admitting, let short replies finish, resubmit the rest elsewhere

Background work

  • Batch, evaluations, flex-tier traffic in the troughs
  • First to pause when interactive load rises

Deep Dive 4: Streaming Tokens Reliably, and Assembling the Context (Week 3 Concepts: Backpressure, Week 2: Idempotency)#

You

"Each reply is a few hundred small events over 10 to 30 seconds, to phones that change networks and users who press Stop. The stream must never make the GPU repeat work, or keep it working for nobody."

Your turn · checkpoint 6not answered

Wi-Fi to 4G at token 212

Meera walks out of a café while her reply is streaming. Her phone drops Wi-Fi at token 212 and reconnects on mobile data two seconds later, to a different edge server. Then she taps Stop at token 260. What does she see, what does the GPU do, and what is saved in her conversation?

Think it through, or jot it down. Then compare.

One strong answer

  • Number every event: each token goes out as a server-sent event with an id; on reconnect the app sends Last-Event-ID: 212, which any edge server can forward.
  • Keep a short log per reply (in memory, replicated, kept for minutes): the new connection gets tokens 213 onwards from the log, then goes live. The GPU never noticed.
  • Stop means stop: the cancel aborts the sequence and the replica frees its KV blocks at the next step. No reconnect within ~30 s counts as a cancel too.
  • Partial replies are kept: the 260 tokens are stored, marked "stopped", so the next turn sees what she saw.
  • Retries are idempotent: the message carries a client key, so a resent "send" attaches to the reply already being written.

The Challenge:

How streaming goes wrong
  • A dropped connection makes the app resend the message, and the GPU writes the whole reply again
  • A closed tab leaves a sequence generating 2,000 tokens for nobody, holding 0.66 GB of KV cache
  • A slow client's unsent events pile up in a server's memory
  • Safety checks that wait for the full reply destroy time to first token

The Solution:

OpenAI's public API shows a practical shape: responses stream as server-sent events with typed events (response.created, response.output_text.delta, response.completed, error), and its "background mode" lets a client cancel a response or resume a stream from the last event's sequence number (starting_after).

DiagramResume and cancel a stream
%% title: Resume and cancel a stream
sequenceDiagram
  autonumber
  participant A as App
  participant E as Edge server
  participant R as Relay + token log
  participant G as Replica
  A->>E: Send message, open stream
  E->>R: Subscribe to reply r88 from the start
  R->>G: Generate r88
  loop each token
    G-)R: Token n, appended to r88's log
    R-)A: Event id n, through the edge
  end
  Note over A,E: Wi-Fi drops at token 212
  A->>E: Reopen, Last-Event-ID 212 (another server)
  E->>R: Subscribe to r88 after 212
  R-)A: Tokens 213 to 240 from the log, then live
  A->>E: Tap Stop at token 260
  E->>R: Cancel r88
  R->>G: Abort r88, free its KV blocks
  R->>R: Save 260 tokens, marked stopped
pythonsse · 41 lines
# relay/stream.py

"""SSE for one reply, with resume and cancel (illustrative, asyncio). The
token log is per reply, replicated in memory for minutes; any edge can serve it."""

import asyncio

IDLE_CANCEL_S = 30          # no client attached this long: stop generating
SEND_TIMEOUT_S = 10         # a client this slow is dropped; it can resume


def sse(event_id: int, data: str, event: str = "delta") -> bytes:
    lines = "".join(f"data: {line}\n" for line in data.split("\n"))
    return f"id: {event_id}\nevent: {event}\n{lines}\n".encode()


async def stream_reply(reply_id: str, last_event_id: int, log, conn) -> None:
    """Serve tokens after last_event_id, from the log, then live."""
    log.attach(reply_id)
    try:
        cursor = last_event_id
        async for n, token, final in log.read_from(reply_id, after=cursor):
            try:
                await asyncio.wait_for(conn.send(sse(n, token)), SEND_TIMEOUT_S)
            except asyncio.TimeoutError:
                return                                    # slow reader: it reconnects with Last-Event-ID
            cursor = n
            if final:
                await conn.send(sse(n + 1, final, event="done"))   # finished, stopped or failed
                return
    finally:
        log.detach(reply_id)          # if nobody re-attaches within IDLE_CANCEL_S, cancel()


async def cancel(reply_id: str, log, engine, store, reason: str) -> None:
    """User pressed Stop, or nobody is listening. Safe to call twice."""
    if not log.mark_final(reply_id, reason):
        return                                            # already finished or cancelled
    await engine.abort(reply_id)                          # blocks freed at the next step
    text = log.text(reply_id)
    await store.save_partial(reply_id, text, status=reason)   # the next turn sees what the user saw

Conversation Storage and Context Assembly#

WorksheetConversation data (assumptions labelled)
  • Per exchange: ~100 tokens in + ~500 out ≈ 600 tokens × 4 bytes ≈ 2.4 KB of text
  • 4.3 billion a day × 2.4 KB ≈ 10 TB a day, ~3.8 PB a year before compression and replication
  • Reads: each message loads its conversation's recent turns, ~50,000 a second, from a cache of active conversations

Context assembly happens in the conversation service, before the router:

  • In a fixed order, for the cache: system instructions, tools, saved user memory, the conversation, then the new message. Whatever changes per request goes last, so everything before it is a reusable prefix.
  • Within a budget: past the model's context, drop or summarise the oldest turns, never the system instructions.
  • Count tokens with the model's tokenizer before routing: quotas, routing and admission are all in tokens.

Safety in the stream:

  • Input: a fast classifier on the new message, before generation, on its own small-model GPUs; it adds tens of milliseconds to time to first token
  • Output: a classifier on each sentence or ~100-token window, alongside the stream. If it flags, the relay stops the reply and the app replaces it with a notice; the log knows which tokens were shown.
Quick check4 questions · +5 XP each

Routing and streaming

  1. Q1/4 A paid user's turn 6 arrives. The replica holding turns 1 to 5 has a queue 3× its normal length. What should the router do?

    Answer & explanation

    Answer: B

    Cache affinity is a preference, not a rule. When the warm replica would break the TTFT target, recomputing the prefix elsewhere is cheaper than the wait.

  2. Q2/4 Why are quotas charged in tokens rather than requests?

    Answer & explanation

    Answer: C

    A 100-token question and a 100,000-token document are both "one request" but differ enormously in prefill, KV cache and cost.

  3. Q3/4 Why must a new GPU replica be planned for ahead of the evening peak?

    Answer & explanation

    Answer: A

    Machine, image, 140 GB of weights and warm-up add up to several minutes; forecasting the daily wave beats reacting to the queue.

  4. Q4/4 A user closes the tab mid-reply and never reconnects. What should happen?

    Answer & explanation

    Answer: C

    Generating for nobody wastes GPU steps and holds memory other users need. Save the partial reply and stop.

Deep Dive 5: Overload and Failure (Week 2 Concepts: Failure-First Design, Week 10: Incident Management)#

You

"Three bad days: more demand than GPUs, a GPU dying mid-reply, a region going away. Each time the system decides who gets served at what quality, and tells people the truth."

Your turn · checkpoint 7not answered

A replica dies with 225 replies in flight

At 21:12 one GPU in a 4-GPU replica fails. All 225 replies on it stop mid-sentence, some at token 3, some at token 480. How do you recover them so users barely notice?

Think it through, or jot it down. Then compare.

One strong answer

  • Detect fast: no tokens for ~2 s on every stream from that replica, and a failed health check. The router stops sending it work; the fleet manager replaces it.
  • Nothing sent is lost: each reply's context and tokens so far are in the token log.
  • Resubmit as continuations: context + tokens already streamed, at the front of its lane. Another replica prefills that (~0.15 s for 2,000 tokens) and continues from the next token.
  • What the user sees: a pause of a second or two, then the reply carries on. What follows the pause is a fresh sample, which is fine because nobody has seen it.
  • Capacity: 225 resubmissions are a small prefill burst. A rack or power failure is not; that needs the overload ladder.
DiagramRecovering replies from a failed replica
%% title: Recovering replies from a failed replica
flowchart TD
  FAIL(["Replica r17 fails<br/>225 replies mid-stream"]):::bad
  DET["Relay: no tokens for 2 s<br/>health check confirms"]:::service
  LOG[("Token log<br/>context + tokens sent")]:::store
  RES["Resubmit each as a continuation<br/>context + partial reply"]:::core
  LANE[["Front of its tier lane"]]:::queue
  NEW["Another replica<br/>prefill, then decode on"]:::service
  USER(["User sees a short pause<br/>then the reply carries on"]):::ok
  DRAIN["Fleet manager<br/>drain and replace r17"]:::service
  FAIL --> DET --> RES
  LOG --> RES --> LANE --> NEW --> USER
  DET -.-> DRAIN

A Region Goes Down#

WorksheetLosing one region of four (illustrative)
  • Run each region at most 75% full at peak: four carry the load of three, so losing one leaves three at 100%
  • In-flight replies are lost with the region unless the token log is replicated out of it; the honest default is "Connection lost. Regenerate?" with the partial text kept
  • Cold prefix caches: if half the input tokens were cached, the moved traffic needs about 2× the prefill until caches warm, and that capacity comes from the degradation ladder, not spare GPUs

The Degradation Ladder#

Under overload, cheapest first
  1. Pause background work
    • Batch, evaluations and flex traffic stop admitting; their GPUs go to interactive lanes in seconds
  2. Shorten what is optional
    • Lower free-tier maximum output; send borderline requests to the fast model, not the reasoning one
  3. Fall back to the smaller model
    • Free users past their limit, then all free users, get the mini model, at a fraction of the GPUs per token, and are told so
  4. Queue with an honest ETA
    • A bounded free-tier queue showing a time ("about 40 seconds") from queue depth and throughput (Little's law)
  5. Shed free-tier work with a clear message
    • "ChatGPT is at capacity for free accounts. Try again in about 10 minutes." Paid targets hold throughout
What has happened in public
  • Image launch, March 2025: Sam Altman posted "our GPUs are melting", announcing temporary image rate limits and three generations a day for free users; days later ChatGPT added a million users in an hour
  • 11 December 2024: from 3:16 to 7:38 pm PST all OpenAI services were degraded or down. A new telemetry service made every node run Kubernetes API operations whose cost grew with cluster size, overwhelming the control plane of large clusters; cached DNS records hid the damage at first, and rolling back needed the overwhelmed control plane. Follow-ups included phased rollouts, break-glass access and decoupling the data plane from the control plane
  • The lesson: the serving path must keep working on what it already knows when the control plane is unavailable
Quick check4 questions · +5 XP each

Overload and failure

  1. Q1/4 A replica fails with replies in flight. Why can they continue on another replica without regenerating what users saw?

    Answer & explanation

    Answer: B

    A new replica prefills context + partial reply and continues from the next token; what the user already read never changes.

  2. Q2/4 Four regions each run at 75% of capacity at peak. One fails. What load do the other three face?

    Answer & explanation

    Answer: C

    Total load is 4 × 75% = 300% of one region; spread over three regions, that is 100% each, before the cold-cache prefill.

  3. Q3/4 Under overload, which step comes first on the degradation ladder?

    Answer & explanation

    Answer: A

    Background work is the cheapest to delay; its GPUs return to interactive lanes in seconds, before any user notices.

  4. Q4/4 What made the December 2024 outage slow to fix?

    Answer & explanation

    Answer: D

    OpenAI's postmortem describes being locked out of the Kubernetes API servers that the faulty telemetry service was overloading.

Phase 5 · Stage 5 of 7 · 5 min budget · 1 min read

Phase 5: Scaling and Edge Cases (5 minutes)#

Interviewer

"Walk me through a day on the fleet, then the cases that bite."

A Day on the Fleet#

A day on the fleet (UTC; the shape is illustrative)
  • 00:00-06:00 · Asia's day, the Americas' night
    • American regions run background work in the trough
  • 06:00-12:00 · Europe wakes up
    • Capacity placed ahead of the European morning; spot added for the peak
  • 13:00-17:00 · The overlap
    • Europe's afternoon, the Americas' morning, India's evening: the day's peak; background work paused
  • 17:00-23:00 · The Americas' afternoon
    • Asian capacity back to background work; rollouts start in the quietest region

Edge Cases#

Edge cases
Edge case 1: A 100,000-token document

Problem: One prompt needs ~33 GB of KV cache in bf16 and at least 7 seconds of a whole replica's compute to prefill (2 × 70 billion × 100,000 FLOPs, before attention), delaying everyone behind it.

Solution: Chunked prefill so decode keeps running; a pool sized for long contexts; quotas in tokens.

Edge case 2: "Regenerate" pressed five times

Problem: Each press is a full new reply.

Solution: The prefix is cached, so it costs decode only; one reply in flight per conversation cancels the previous attempt.

Edge case 3: A model upgrade

Problem: A new version must replace the old on thousands of replicas without dropping replies.

Solution: Drain (no new admissions, let replies finish), reload, warm, and shift traffic by percentage, one region first.

Quick check3 questions · +5 XP each

Scaling and edge cases

  1. Q1/3 A 100,000-token prompt arrives. Which measure protects the other users on that replica?

    Answer & explanation

    Answer: B

    Splitting the prefill into chunks keeps other replies streaming instead of stalling for the whole prompt.

  2. Q2/3 Why does pressing "Regenerate" cost mostly decode, not prefill?

    Answer & explanation

    Answer: B

    The same context was just prefilled on that replica; only the new reply's tokens need computing.

  3. Q3/3 How do you roll out a new model version without dropping replies?

    Answer & explanation

    Answer: B

    Draining lets in-flight replies finish; shifting by percentage limits the blast radius of a bad version.

Phase 6 · Stage 6 of 7 · 2 min

Phase 6: Monitoring and Operations#

You

"Users feel two latencies and a stall; finance feels cost per token. The dashboard shows both, per tier and region."

Monitoring Dashboard#

Fleet dashboard — 16:30 UTC (illustrative)
Latency, by tier
Metric Paid Free
Time to first token p50 / p95 0.42 / 0.88 s 0.61 / 1.70 s
Time per output token p50 / p95 31 / 44 ms 33 / 48 ms
Streams that stalled > 2 s, last hour 0.03% 0.05%
Queue depth / est. wait 0 / 0 s 2,300 / 0.04 s
GPUs and caches
  • Messages 68,000/s; output tokens 34 million/s; streams open 1.2 million
  • HBM bandwidth in use 61%; KV cache occupancy 88%; preemptions 140/s
  • Prefix cache hit rate 52% of input tokens; cancellations freeing blocks 3,900/s
  • Cost $1.90 per million output tokens today; background lane paused since 15:10
Alerts (today)
  • 15:10 (amber) — Europe-West KV occupancy over 90% for 5 minutes, background lane paused
  • 15:42 (amber) — Replica eu-w-2210 failed, 218 replies resumed elsewhere, median pause 1.4 s

SLOs for the Platform#

SLOs for LLM serving
SLO 1: First token
  • Target: TTFT p95 < 1 s for paid and < 2 s for free, in every 5-minute window, per region
SLO 2: Smooth streaming
  • Target: TPOT p95 < 50 ms; 99.9% of streams with no pause over 2 s
SLO 3: Replies delivered
  • Target: 99.9% of messages end in a complete reply or an actionable error per 30 days; the error budget is 0.1%, about 43 minutes of total outage equivalent
SLO 4: Cost
  • Target: GPU cost per million output tokens tracked daily, beside the utilisation and prefix hit rate behind it
Quick check3 questions · +5 XP each

Reading the dashboard

  1. Q1/3 The free tier shows 2,300 queued requests and an estimated wait of 0.04 s. Which law gives that wait?

    Answer & explanation

    Answer: A

    Free users send ~85% of 68,000 messages a second, about 58,000; 2,300 queued ÷ 58,000 a second ≈ 40 ms of waiting.

  2. Q2/3 Prefix cache hit rate falls from 52% to 20% after a deploy. What do you expect next?

    Answer & explanation

    Answer: C

    Missed prefixes must be recomputed; a change that reorders the system instructions can do this.

  3. Q3/3 Why is "streams that stalled more than 2 s" on the dashboard when p95 TPOT is fine?

    Answer & explanation

    Answer: B

    A replica failure or a long prefill can freeze replies for seconds while per-token percentiles barely move.

Final challenge · Stage 7 of 7 · 2 min

Final Challenge: Launch Day, and Demand Is 5× Capacity#

The interviewer opens a launch calendar. "One more."

Interviewer

"Tomorrow at 17:00 UTC a new image feature launches to every user, on a pool of 8,000 GPUs. It goes viral: demand runs at five times what the pool can serve, while chat is still near its daily peak. Decide who gets served, at what quality, and what you tell people."

The launch-day brief
  • Image pool: 8,000 GPUs; one full-quality image takes 16 GPU-seconds (assumption)
  • A "fast" mode at lower resolution takes 8 GPU-seconds (assumption)
  • Demand: 5× capacity; paid users send 20% of image requests
  • Up to 4,000 more GPUs can be freed from background work (batch, evaluations, flex) in about 10 minutes
  • Chat is near its daily peak; its latency targets still apply
Your turn · checkpoint 8not answered

Who gets an image, at what quality, and what do they see?

Work out capacity and demand, what the extra GPUs and fast mode buy, the paid/free split, and what each user sees.

Think it through, or jot it down. Then compare.

One strong answer

  • Capacity: 8,000 GPUs ÷ 16 GPU-s = 500 images a second. Demand at 5× = 2,500 a second: 500 paid, 2,000 free.
  • A queue can't absorb it: 2,000 a second of excess is 7.2 million waiting after an hour.
  • Add what is cheap: pause background work, freeing 4,000 GPUs in ~10 minutes → 12,000 GPUs. Not chat's GPUs: it is near its peak.
  • Paid at full quality: 500 a second × 16 GPU-s = 8,000 GPUs, with one image in flight per user.
  • Free in fast mode: the remaining 4,000 GPUs ÷ 8 GPU-s = 500 a second, a quarter of free demand. Free accounts get a small daily allowance (OpenAI announced three a day in March 2025) and a bounded queue with an honest ETA; past that, "at capacity, back at 19:00 UTC".
  • Tell people before they wait: a banner ("Very high demand: free accounts get 3 images today, in fast mode"), an ETA on each request, the remaining allowance. Never "something went wrong".
  • Protect chat: separate pools and lanes, so image demand can't touch chat's time to first token.
Quick check4 questions · +5 XP each

Final challenge · Quest check

  1. Q1/4 An image pool has 8,000 GPUs and each image takes 16 GPU-seconds. How many images a second can it make?

    Answer & explanation

    Answer: B

    8,000 GPU-seconds a second ÷ 16 GPU-seconds an image = 500 images a second.

  2. Q2/4 Which extra GPUs should launch day take first?

    Answer & explanation

    Answer: A

    Background work can wait hours without breaking a promise; chat's latency targets can't.

  3. Q3/4 Free demand is four times what is left for it. What should free users get?

    Answer & explanation

    Answer: D

    A smaller cost per image, a quota and a truthful wait keep the queue bounded and expectations right.

  4. Q4/4 Why keep image and chat on separate pools and lanes on launch day?

    Answer & explanation

    Answer: B

    Isolation keeps a launch's surge from breaking the core product for everyone else.

Wrap-up · 6 min

Interview Conclusion#

Transcript
Interviewer

"Why not give every user their own GPU slot, so nobody waits for anyone?"

You

"Decode is memory-bound. One user on a replica gets ~95 tokens a second and leaves 99.7% of the compute idle; 225 users get ~46 each and the replica writes 110 times more. Sharing is the only way the price works; the craft is making it invisible."

Interviewer

"Where would you spend your next engineering quarter?"

You

"On the prefix cache hit rate. Every point of it is prefill we don't buy: per-request content at the end of the prompt, conversations routed to the replica holding their history, blocks kept for minutes. Then a weekly overload drill: replay a launch-day curve against a staging region, kill a replica mid-stream, and measure time to first token and resumed streams."

Interviewer

"The single most important decision?"

You

"Scheduling by tokens and memory, not requests. Once admission, quotas, routing and autoscaling count tokens and KV blocks, overload becomes a choice about who gets which model and when, instead of a spinner for everyone."

Summary: Concepts Applied from 10-Week Course#

Concepts from the 10-week course in LLM serving
  • Week 0: Foundations
    • Back of the envelope — tokens a second, KV bytes per token, GPUs and megawatts
    • Operating systems — paged KV memory, like virtual-memory pages
  • Week 1: Data at scale
    • Rate limiting — token buckets in tokens, per user and per model
    • Hot keys — a popular prefix overflowing beyond one replica
  • Week 2: Failure-first design
    • Idempotency — a resent message attaches to the reply already being written; a silent stream resumes elsewhere
  • Week 3: Messaging and async processing
    • Backpressure — bounded queues per lane, slow readers dropped and resumed
  • Week 4: Caching
    • Prefix caching — shared instructions and conversation history reused across turns
  • Week 9: Multi-tenancy
    • Noisy neighbours — tier lanes and per-user fairness inside a lane
  • Week 10: Production readiness
    • Capacity planning — forecast the daily wave, N+1 regions, a degradation ladder

Why ChatGPT Matters#

Why this is a different kind of hard problem
The unit of work is a token, not a request
  • A reply's length, memory and cost are unknown until it ends, so every layer schedules work it can't measure in advance
The scarce resource is GPU memory
  • After the weights, the KV cache decides how many people one replica serves; batching, paging and prefix caching fit more people into the same bytes
Fairness is a product decision
  • When demand beats capacity, someone gets a smaller model, a wait or a "no"; a good system chooses deliberately, and says so

"A reply is a few hundred steps on a shared GPU. The craft is making each step serve as many people as possible, and making sure none of them can tell."

Self-Assessment Checklist#

After studying this case study, you should be able to:

Estimation:

Estimation0/2 ready
  • Turn messages a day into tokens a second, and tokens a second into GPUs and power
  • Compute KV cache bytes per token from a model's layers, KV heads and head dimension

Inference:

Inference0/2 ready
  • Explain why decode is memory-bound and why batching multiplies throughput
  • Compare static and continuous batching, and explain paged KV memory and prefix caching

Fleet:

Fleet0/3 ready
  • Route by model, region and prefix affinity with overflow
  • Design tier lanes, token-based quotas and admission control
  • Plan GPU capacity around minutes-long cold starts and spot interruptions

Reliability:

Reliability0/2 ready
  • Stream with resumable server-sent events and cancellation that frees GPU memory
  • Recover replies from a failed replica, size regions for N+1, and write the degradation ladder

Sources#

OpenAI announcements and statements:

OpenAI API documentation:

Papers:

Hardware, models and cloud:

Design note: the architecture, code, thresholds, rates and dashboards here are an illustrative design, sized with an open 70B model on H100 GPUs, not a description of how OpenAI serves ChatGPT beyond what OpenAI has published; the real fleet's models and hardware differ.

Further Reading#

Related systems to study:

  • Read

    IRCTC Tatkal (Bonus Problem 11): admission control when demand far exceeds supply

  • Read

    Netflix (Bonus Problem 6): streaming bytes to millions at once

  • Read

    Zerodha Kite (Bonus Problem 13): honest states and a fixed degradation ladder under a spike

Foundational concepts (this course):

  • Read

    Week 0, Part 3 — Back of the Envelope Estimation

  • Read

    Week 1, Day 3 — Rate Limiting at Scale

  • Read

    Week 3, Day 3 — Backpressure and Flow Control

  • Read

    Week 4, Day 3 — Thundering Herd

  • Read

    Week 9, Day 2 — Noisy Neighbour

  • Read

    Week 10, Day 4 — Capacity Planning

"Every reply is a few hundred steps on a GPU somebody else is also using. Serve it well and nobody notices the sharing; serve it badly and everybody notices the wait."

End of Side quest 14 · ChatGPT LLM Serving
Clear the questFinish ChatGPT.Stages clear as you read them. Marking complete unlocks after ~7 more min of reading.
  • 0/7stages cleared
design sheet · 7 diagrams · pdfRevise the whole design in ten minutesThe numbers, architecture, sequence, class and state diagrams, decisions and failure modes of this quest on a few printable pages.open the sheet →

Go deeper · reading trail

all reading →

Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.

UC BerkeleyPaper · ~45 min

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li et al. · 2023
What
The vLLM paper from UC Berkeley, which introduced PagedAttention.
Key idea
The KV cache is stored in fixed-size blocks mapped like virtual memory pages, which cuts fragmentation and lets requests share prefixes, raising throughput.
Why here
The paged KV memory and prefix caching deep dive of this quest at its source.
taught here: Paged KV memory
Seoul National UniversityPaper · ~40 min

Orca: A Distributed Serving System for Transformer-Based Generative Models

Gyeong-In Yu, Joo Seong Jeong et al. · 2022
What
The paper that introduced iteration-level scheduling for LLM serving.
Key idea
Instead of waiting for a whole batch to finish, the scheduler adds and removes requests at every decoding step, so short answers do not wait for long ones.
Why here
Where this quest's batching deep dive comes from: continuous batching is iteration-level scheduling.
taught here: Continuous batching
AnyscaleBlog post · ~18 min

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

Cade Daniel, Chen Shen et al. · 2023
What
A benchmark-driven explainer of LLM inference batching.
Key idea
LLM serving is memory-bound, so batching is the main lever; continuous batching and paged memory raised throughput up to 23x in their tests while lowering median latency.
Why here
The back-of-the-envelope GPU maths of this quest checked against real measurements.
taught here: Throughput vs latency
Up next
SIDE QUEST 15Figma Multiplayer EditingStart →

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…