Day 1LLM Serving: Tokens, Latency and Cost

Size it in tokens, not requests. A model call's work, price and rate limits scale with tokens in and out. 900 turns a minute sounds small; at ~3,835 input tokens a turn it's 3.45M input tokens a minute.

Prefill, then decode. Prefill reads every input token in one parallel, compute-bound pass and sets time to first token. Decode writes one token per step, bound by memory bandwidth, and sets time per output token.

Output length is the latency lever. Latency = TTFT + TPOT × output tokens. A 250-token reply at 20 ms a token is ~5 s of decode. Halving output may cut ~50% of latency; halving the prompt only 1-5%.

Tokens resident set concurrency. The KV cache holds every token of every running sequence in GPU memory. A 20,000-token prompt holds the memory of five 4,000-token ones, so long prompts cut concurrency and raise TTFT.

Errors arrive after the 200. Stream end to end: no buffering proxies, SSE comment heartbeats under idle timeouts, cancel on disconnect. An overloaded_error can arrive mid-stream: tell the client in-band, store the reply as interrupted.

Admit by tokens in a gateway. One gateway counts tokens, keeps shared buckets for input and output, reserves a share for interactive chat, queues with deadlines, and retries 429 and 529 only before the first byte, with retry-after and jitter.

• Sizing by requests: 900 a minute is 3.45M input tokens a minute.
• Treating 200 as success, or saving a partial reply as the answer.
• Retrying 429s at once, after the first byte, or on a spend-cap 429.

Day 2Embeddings and Vector Search

Keywords miss meaning, vectors miss tokens. "paisa wapas" needs an embedding to find the refunds article; "RTN-4471" needs BM25, because one return code embeds much like another. Use the same embedding model and version for queries and documents.

Exact search is a bandwidth problem. 3M vectors × 4 KB is 12 GB per query, ~0.6 s at 20 GB/s. Keep exact search for ground truth and selective filters: one customer's 2,000 tickets scan faster than any index.

HNSW, a graph you descend. Sparse top layers make long greedy jumps; layer 0 holds every vector and keeps a candidate list of size ef. Raising ef_search buys recall with latency. Deleted rows linger until a vacuum or rebuild.

IVF and product quantisation. IVF probes nprobe of nlist k-means clusters; cheap appends, but centroids drift. PQ stores a 1,024-dim vector as 64 one-byte codes, 64× smaller; rescore the top 100-200 with full vectors.

Filter inside the search. pgvector post-filters approximate scans: ef_search 40 at 10% selectivity returns about 4 rows, silently. Use iterative scans, partial indexes for a few known values, or an exact scan for tiny sets.

Fuse ranks, not scores. BM25 is unbounded and query-dependent; cosine sits in a narrow band. Reciprocal rank fusion sums 1 ÷ (60 + rank) per retriever, so agreement wins. Pass the reranker a generous candidate list.

• Vector search alone, so order codes and SKUs go missing.
• Filtering after the index and getting 3 results instead of 10, with no error.
• Querying with a new embedding model against vectors from the old one.

Day 3RAG Systems

The answer is only as fresh as the index. RAG is a search pipeline with a model at the end. Lewis et al. swapped 2016 and 2018 indexes and accuracy collapsed; MegaMart's stale policy was a hot-swap that never happened.

Chunk along structure. Parse tables with their headings, chunk one section at a time with document and section titles inside, and dedup by content hash. Contextual retrieval prepends 50-100 tokens of document context to each chunk.

Freshness is an SLO. CDC keyed by doc_id, idempotent upserts on doc_id:version:chunk_no, deletes as events, a dead-letter queue for poison documents, a reconciliation sweep, and a canary measuring publish to searchable.

Retrieve wide, rerank narrow. Hybrid search returns ~150 candidates; a cross-encoder reads query and chunk together and keeps ~8. Below a score threshold, abstain and offer a human rather than answer from weak context.

Retrieval runs as the user. Filter by tenant and ACL inside retrieval, so unauthorised chunks never reach the reranker, the model or the logs. A prompt saying "don't reveal" isn't access control. Entitlements down: public content only.

• Tuning the prompt before reading the retrieved chunks.
• Fixed-size chunks that cut tables from their headings.
• Permissions in the prompt, or a filter applied after the top-k.

Day 4Evaluation and Guardrails

Quality is measured, not unit-tested. Outputs vary between runs, every prompt change is global, and wrong answers look like successes. Gate changes on a versioned golden set with never-regress cases, scored per slice of intent and language.

Graders, cheapest first. Code checks facts, citations and behaviour. An LLM judge handles faithfulness and preference. Humans check the judge: accept it only if its kappa with people approaches the people's kappa with each other.

Judges are biased instruments. Position, verbosity and self-preference. Compare in both orders and count a win only if it survives the swap; judge with a different model; give a reference; use binary, checkable rubrics.

Faithful isn't correct. Score retrieval and generation separately. Smaller chunks dropped context recall from 0.91 to 0.78 while faithfulness rose: answers stuck to thinner context. The stale 30-day policy scored perfect faithfulness too.

Assume the model will be fooled. Injection arrives in seller policies, uploads and tool results, and no filter catches it reliably. So limit what a fooled model can do: narrow tools, server-side checks, confirmation for side effects.

Break the lethal trifecta. Private data plus untrusted content plus an outbound channel can leak data. The first two are the product, so remove the third: allow-listed links and images, email_return_label(order_id) instead of send_email.

• Shipping on a vibe check of 15 hand-picked chats.
• A classifier as the injection defence while the model picks amounts and recipients.
• A fallback model nobody ran against the golden set.

Day 5Cost, Caching and Operations

Four meters. Fresh input, cache write, cache read and output, each priced per model. History is resent every turn, so cost grows with the square of the turns. Compare models on cost per resolved conversation.

Prompt caching is cost and capacity. An exact prefix, built tools, system, then messages. A 5-minute write is 1.25× input, a read 0.1× or less, and for most Claude models reads don't count toward ITPM. Stable first, volatile last.

Semantic caches skip the guardrails. A false hit is a confident wrong answer that never met the model or Day 4's checks. Only general, non-personal questions; keys scoped by tenant, language and prompt version; purge from sources; measure false hits.

Route easy turns to a small model. A router decides before the call, a cascade after scoring the cheap answer. At $0.0012 vs $0.022 a conversation, a cascade pays until ~95% escalate. Keep each conversation on one model.

Budgets that degrade. Reserve the worst case from max_tokens, settle from usage, per feature, tenant, user and conversation. At 80%, thrifty; at 95%, minimal; spent, hand off. A step cap per turn stops loops first.

Fail over in steps. One retry layer, in the gateway, with SDK retries off. The fallback's cache is cold, so a conversation costs ~3× until it warms. Shift 10%, 25%, 50% to avoid its acceleration limits.

• A timestamp on line one of the system prompt, so every request misses the cache.
• Charting Anthropic's input_tokens alone, which excludes cached tokens.
• One organisation-wide budget, so a runaway loop takes the chat down.

BossDesign an AI Customer-Support Assistant

The model proposes, code decides. The model returns tool_use blocks; a deterministic controller runs the agent loop, checks every call in a tool gate, and decides. Ownership, amounts, limits, confirmation and idempotency live outside the model.

The safest argument doesn't exist. issue_refund takes only an order id: the gate computes the amount with refund_policy.allowed_amount(), over ₹5,000 hands off. email_return_label(order_id) lets the server pick the recipient. No free text leaves.

Proposals, confirmed and executed once. A side effect becomes a proposal row the customer confirms with a button showing the server's action. Confirming re-checks it and executes under an idempotency key from the proposal id, so double taps refund once.

Untrusted text only in tool results. Seller policies arrive inside lookup_order's result, JSON-encoded and labelled. The turn is then tainted, so any side effect needs confirmation. Where text sits lowers the odds; the gate decides safety.

A step cap, then instructive errors. Five tool calls per turn, then an is_error result telling the model to offer a human; hand off if it keeps going. Errors say what to do next, not just what failed.

Keep the tools array identical. For answers without actions, don't remove tools: they're the first cached segment, so every request would miss and hit 429s. The gate denies side effects instead, and pending proposals fail closed.

• issue_refund(order_id, amount) executed as soon as the model calls it.
• Removing tools from the request during an incident, breaking the cached prefix.
• Letting a continuation replay a side effect instead of the confirm endpoint.