ChatGPT LLM Serving, the whole design.

%% title: LLM serving on one page: everything keeps the GPUs busy and fair
flowchart TD
  APP@{ icon: "sd:phone", label: "ChatGPT apps · web, mobile, desktop<br/>API clients" } ==> GW@{ icon: "sd:gateway", label: "Edge + API gateway<br/>TLS · SSE streams · auth<br/>token quotas per user per model" }
  GW ==>|"message"| CONV["Conversation service<br/>context in cache-friendly order<br/>count tokens"]:::service
  CONV --> HIST@{ icon: "sd:database", label: "Conversation store<br/>~10 TB a day · active cache" }
  CONV ==>|"context"| SAFE@{ icon: "sd:shield", label: "Safety classifiers<br/>input before · output per sentence" }
  SAFE ==>|"generate"| ROUTE["Router<br/>model · region · prefix affinity + overflow"]:::core
  ROUTE --> MINI["Mini-model pool<br/>fallback past limits"]:::service
  ROUTE ==> LANES@{ icon: "sd:queue", label: "Tier lanes<br/>paid · free · background" }
  LANES ==> REP["Model replicas · 4 × H100<br/>continuous batching · paged KV<br/>prefix cache"]:::hot
  REP ==>|"tokens"| TLOG@{ icon: "sd:cache", label: "Relay + token log<br/>numbered events · kept minutes<br/>streams back through the edge" }
  REP --> USE@{ icon: "sd:stream", label: "Usage + telemetry<br/>limits · billing · TTFT · cost" }
  USE --> FLEET@{ icon: "tech:kubernetes", label: "Fleet manager · Kubernetes<br/>forecast capacity · spot<br/>adds, drains replicas" }
  class APP client
  class GW edge
  class HIST store
  class SAFE,TLOG cache
  class LANES,USE queue
  class FLEET service

%% title: Inside one replica: one step for the whole batch
flowchart TD
  W[["Waiting · from the tier lanes"]]:::queue --> ADM{"Prompt blocks fit,<br/>above the reserve?"}:::service
  ADM -->|"yes · reuse cached prefix blocks"| RUN["Running batch · ~225 seqs<br/>KV in 16-token blocks"]:::core
  ADM -->|"no"| W
  RUN --> STEP["One forward step<br/>prefill arrivals · 1 token for the rest<br/>reads all weights once"]:::hot
  STEP --> END{"Stop token, max tokens<br/>or cancelled?"}:::service
  END -->|"no · append K, V"| RUN
  END -->|"yes"| FREE(["Leave next step<br/>blocks freed, prefix kept"]):::ok
  RUN -.->|"out of blocks"| PRE(["Preempt the newest<br/>requeue, recompute later"]):::bad

%% title: One message, keyboard to screen
sequenceDiagram
  autonumber
  participant A as Meera's app
  participant E as Edge + gateway
  participant C as Conversation svc
  participant S as Router
  participant M as Replica r17
  A->>E: send (conversation c42, key m9)
  E->>E: auth · token quota ok (paid)
  E->>C: forward · open SSE stream
  C->>C: input check · 2,000-token context
  C->>S: generate (model, tier, prefix hash)
  S->>M: admit · prefix already cached here
  M->>M: prefill only ~300 new tokens
  M-)A: token 1 at ~0.42 s (via relay, edge)
  loop every ~35 ms until a stop token
    M-)A: next token · numbered event
  end
  M->>M: free KV blocks
  C->>C: output check · save reply
  C-)A: done
  Note over M: r17 writes ~224 other replies in the same steps

%% title: Wi-Fi drops at token 212, then Stop at 260
sequenceDiagram
  autonumber
  participant A as App
  participant E as Edge server
  participant R as Relay + token log
  participant G as Replica
  A->>E: send message, open stream
  E->>R: subscribe to reply r88
  R->>G: generate r88
  loop each token
    G-)R: token n, appended to r88's log
    R-)A: event id n
  end
  Note over A,E: Wi-Fi drops at token 212
  A->>E: reopen · Last-Event-ID 212 (another edge)
  E->>R: subscribe to r88 after 212
  R-)A: 213 onwards from the log, then live
  Note over G: GPU never noticed
  alt user taps Stop at 260
    A->>E: stop
    E->>R: cancel r88
    R->>G: abort · free KV blocks next step
    R->>R: save 260 tokens, marked stopped
  else no reconnect within ~30 s
    R->>G: cancel · nobody is listening
  end

%% title: A replica dies with 225 replies in flight
sequenceDiagram
  autonumber
  participant R as Relay + token log
  participant X as Replica r17
  participant S as Router
  participant N as Replica r42
  participant A as App
  X--xR: GPU fails · tokens stop
  Note over R: no tokens for ~2 s · health check confirms
  R->>S: resubmit each as continuation
  Note over S: context + tokens already sent · front of its lane
  S->>N: admit
  N->>N: prefill ~2,000 tokens (~0.15 s)
  N-)R: next token onwards
  R-)A: reply carries on after a short pause
  S->>S: stop routing to r17 · fleet replaces it

%% title: Core entities
classDiagram
  class Conversation {
    +conversationId : string · partition key
    +userId : string
    +tier : free / paid / business
    +model : string
  }
  class Message {
    +messageId : string
    +clientKey : string · idempotency
    +role : user / assistant
    +text : string · ~2.4 KB a turn
    +status : complete / stopped / flagged
  }
  class Reply {
    +replyId : string
    +replicaId : string
    +lastEventId : int
    +state : ReplyState
  }
  class TokenLogEntry {
    +replyId : string
    +n : int · SSE event id
    +token : string
  }
  class Sequence {
    +replyId : string
    +length : int · tokens so far
    +cachedTokens : int · prefix hits
    +arrived : timestamp · newest preempted first
  }
  class KVBlock {
    +blockId : int · 16 tokens
    +prefixHash : hash · full blocks only
    +refs : int
  }
  Conversation "1" --> "many" Message : turns
  Message "1" --> "1" Reply : answered by
  Reply "1" --> "many" TokenLogEntry : streamed as
  Reply "1" --> "1" Sequence : runs as
  Sequence "many" --> "many" KVBlock : shares prefix blocks

%% title: Reply state machine
stateDiagram-v2
  [*] --> QUEUED: admitted to a tier lane
  QUEUED --> SHED: free tier at capacity
  QUEUED --> RUNNING: blocks fit, prefill
  RUNNING --> PREEMPTED: out of blocks, newest
  PREEMPTED --> QUEUED: requeued, recompute
  RUNNING --> RESUMING: replica failed
  RESUMING --> QUEUED: continuation, front of lane
  RUNNING --> FINISHED: stop token or max tokens
  RUNNING --> STOPPED: user Stop or nobody for 30 s
  RUNNING --> FLAGGED: output check stops it
  FINISHED --> [*]
  STOPPED --> [*]
  FLAGGED --> [*]
  SHED --> [*]
  class FINISHED ok
  class SHED,FLAGGED bad
  class PREEMPTED,RESUMING queue