Cost, Caching and Operations.

Sale week at MegaMart. MegaMart Assist has been live for a month, answering return and refund questions with Day 3's retrieval behind Day 4's guardrails. The answers are good. The bill and the pager are not. (An illustrative scene: the times and amounts are made up; each failure is a common one.)

None of these is a model problem. They're old distributed-systems problems: a cache key that changes on every request, a loop with no limit, a shared quota with no owner, a failover nobody costed, a bill with no labels. The new part is that the dependency is metered by the token, so every mistake lands on the invoice.

Yesterday (Day 4) we measured quality and fenced the assistant in, and those judges, screens and grounding checks are model calls too. Today we make the whole system affordable and operable: what a conversation costs, three kinds of cache, routing easy turns to a cheaper model, batch work at half price, budgets that degrade before they cut off, failover that doesn't blow the budget, the telemetry that says who spent what, and the runbooks for the incidents above.

01Chapter 1: Token Economics#

1.1 The Simple Definition#

An LLM API bills per token, on four meters: fresh input, input written to the provider's cache, input read from that cache, and output. Each meter has its own price on each model. A request's cost is the sum of the four meters times their prices. Everything in this lesson either moves tokens to a cheaper meter or stops sending them.

1.2 The Price List#

Day 1 showed the pattern: output costs about 5× input, and models span 20× or more. Today we also need the cache-write column. Anthropic's list prices as of October 2026, per million tokens (MTok), used as this lesson's example prices:

Two warnings before using any price. Tokens aren't words: Anthropic says its newer tokenizer "produces approximately 30 percent more tokens for the same text than earlier models", and Day 1 showed counts differ by language. And reasoning is output: thinking counts toward max_tokens "even when the thinking content isn't returned" (Anthropic effort docs), and OpenTelemetry counts reasoning tokens inside output tokens. So compare models on cost per resolved conversation, measured from real usage blocks, and keep prices in a dated config table, not in code.

1.3 Per Request, Per Conversation, Per User#

# USD per million tokens: Anthropic list prices as of October 2026 (keep in dated config)
PRICES = {
    "claude-sonnet-5-5": {"in": 2.00, "write_5m": 2.50, "read": 0.10, "out": 10.00},
    "claude-haiku-5-5":  {"in": 0.10, "write_5m": 0.125, "read": 0.01, "out": 0.50},
}   # Haiku 5.5 prompts over 100,000 tokens are billed at higher rates

def request_cost(model: str, usage) -> float:
    p = PRICES[model]
    return (usage.input_tokens * p["in"]                    # only tokens after the last breakpoint
            + usage.cache_creation_input_tokens * p["write_5m"]
            + usage.cache_read_input_tokens * p["read"]
            + usage.output_tokens * p["out"]) / 1_000_000

Day 1's numbers: a ~3,000-token system prompt, ~60-token messages, ~250-token replies and 6 turns give 23,010 input and 1,500 output tokens per conversation, ~$0.061 on Sonnet 5.5, three quarters of it input, because each turn resends everything before it. A user's cost is the sum over their conversations; a feature's is the sum over its calls. Both need labels on every call (Chapter 8).

02Chapter 2: Provider Prompt Caching#

2.1 How It Works#

Day 1 explained that prefill turns every input token into key/value states, the KV cache. Prompt caching keeps those states for a prefix between requests, so the next request that starts with exactly the same prefix skips reprocessing it. From Anthropic's documentation, as of October 2026:

• Order: "Cache prefixes are created in the following order: tools, system, then messages." You mark where a prefix ends with cache_control (up to 4 breakpoints), or use automatic caching, where "the cache point moves forward automatically as conversations grow".
• Exact match: "Cache hits require 100 percent identical prompt segments" up to the marked block, and the prefix must be at least 512 tokens on the 5.5 models.
• Lifetime: 5 minutes, "refreshed for no additional cost each time the cached content is used"; a 1-hour lifetime costs more.
• Price: a 5-minute write is 1.25× input, a 1-hour write 2×, a read 0.1× on most models (0.05× on Opus 5.5 and Sonnet 5.5, 0.025× on Fable 5.1). Caching "pays off after one cache read" at 5 minutes.
• No quality cost: "The response you receive is identical to what you would get if prompt caching were not used."
• Isolation: per workspace; "Different organizations never share caches, even if they use identical prompts."
• Bursts miss: "a cache entry only becomes available after the first response starts", so a thousand identical requests sent at once all pay full price.

%% title: Order the prompt from most stable to most volatile
flowchart TD
  T["1 · Tool definitions<br/>change only on deploys"]:::ok
  SP["2 · System prompt and policies<br/>~3,000 tokens, change only on deploys"]:::ok
  H["3 · Earlier turns of this conversation<br/>written to cache by the previous turn"]:::ok
  BP{{"Breakpoint: everything above is a cache read"}}:::core
  N["4 · This turn only<br/>retrieved passages, current time, the new message"]:::bad
  T --> SP --> H --> BP --> N

The response reports three input fields: cache_read_input_tokens, cache_creation_input_tokens, and input_tokens, which counts only tokens after the last breakpoint. Total input is their sum. A chart of input_tokens alone shows a working cache as a mysterious 90% drop in traffic.

Caching is also capacity. Day 1's calc found that ITPM binds at MegaMart's peak, and for most Claude models "only uncached input tokens count toward your ITPM rate limits". That's why Monday's cache-busting deploy at 10:30 became a rate-limit incident at 20:30.

2.2 The Same Idea at Three Providers#

The advice is the same everywhere: OpenAI says "put stable developer instructions and shared reference material first", Gemini says put "large and common contents at the beginning of your prompt". Two portability traps: on OpenAI models before GPT-5.6, routing to a cache uses a prompt_cache_key, at "about 15 requests per minute" per key; and Gemini's explicit caching is "not supported for Gemini 4 Argon and onward".

03Chapter 3: Application-Level Caches#

A prompt cache still runs the model on every request. An application cache skips the model and returns a stored answer: Week 4's cache-aside with a new kind of key.

3.1 Exact Caching#

The key is a hash of everything that determines the answer: model, parameters, prompt version, messages and retrieved passages. Cloudflare's AI Gateway works this way, serving cached responses "for identical requests", with a TTL, a skip-cache header and a custom-key header (Cloudflare docs). It's safe, but customers rarely phrase things identically, so it hits mostly on machine-made prompts: a classification, a summary of a shared policy page, a re-run.

3.2 Semantic Caching#

A semantic cache embeds the question (Day 2), finds the nearest past question, and returns its stored answer if it's similar enough. GPTCache's modules spell out the parts: an embedding generator, cache storage, a vector store that finds "the K most similar requests", and a similarity evaluator that decides whether it's a hit. Its README names the catch: "In a semantic cache, you may encounter false positives during cache hits and false negatives during cache misses" (GPTCache).

A false hit in a support assistant is a confident, wrong answer that skipped the model and Day 4's guardrails:

The rules follow. Scope the key by tenant, language and prompt version. Cache only non-personal answers: general policy, never "my order". Invalidate from the sources: cached answers are derived data (Day 3), so when a policy page changes, or a customer's data must be erased (Week 9, Day 4), purge every answer built from it (Week 4, Day 2 · Cache Invalidation). Set the threshold from data: grade a sample of hits with Day 4's judge or a human, and raise the threshold until the false-hit rate is acceptable.

3.3 Three Caches Compared#

04Chapter 4: Routing and Cascades#

Most support turns are easy, and "where is my refund?" doesn't need the large model. Two designs use that (Day 4 owns fallback when a model fails; this is choosing a model by difficulty):

• A router decides before the call. A cheap classifier or rules pick the model. RouteLLM, for example, trains routers on human preference data to choose between a stronger and a weaker model (Ong et al., 2024).
• A cascade decides after. Ask the cheap model, score its answer, escalate only if the score is low. FrugalGPT's cascade "sends a query to a list of LLM APIs sequentially. If one LLM API's response is reliable, then its response is returned"; it accepts an answer "if the score is higher than a threshold" and queries the next model otherwise (Chen, Zaharia and Zou, 2023).

%% title: A two-step cascade for one turn
flowchart TD
  Q["Customer turn<br/>with retrieved passages"]:::core
  SM@{ icon: "sd:ml", label: "Small model<br/>Haiku 5.5" }
  SC{"Scorer<br/>grounded? confident?<br/>score ≥ threshold"}
  OK(["Send the small model's answer"]):::ok
  LM@{ icon: "sd:ml", label: "Large model<br/>Sonnet 5.5" }
  OUT(["Send the large model's answer"]):::ok
  Q --> SM --> SC
  SC -->|"yes"| OK
  SC -->|"no: escalate"| LM --> OUT
  class SM,LM external

def answer_turn(convo, threshold: float):
    draft = llm("claude-haiku-5-5", convo, effort="low")
    score = scorer(convo, draft)        # trained and calibrated on Day 4's graded set
    if score >= threshold:
        return draft, "small"
    return llm("claude-sonnet-5-5", convo), "escalated"   # this turn pays for both calls

FrugalGPT reported matching GPT-4 "with up to 98% cost reduction" on its benchmarks in 2023: the paper's result on its tasks, not a promise for yours. Its scorer needs graded examples "from the same or similar distribution as the test examples", which is what Day 4's golden set provides.

Three practical points:

• Keep a conversation on one model. The cache holds a model's own processed prefix, so a turn escalated to another model reprocesses the whole history at full price.
• Lower effort is a third option. Anthropic's effort parameter trades "response thoroughness and token efficiency with a single model"; low gives "significant token savings with some capability reduction". Kept fixed for a conversation, it keeps the cache warm (changing effort mid-conversation invalidates cached messages, unless you use the beta per-message effort setting); test it on the same eval set.
• Alert on the escalation rate. If it creeps from 30% to 70%, the questions or the small model changed, and so did your bill.

05Chapter 5: Batch APIs for Offline Work#

MegaMart's nightly job tags every closed conversation (reason, resolved or not, sentiment) for the support team: 60,000 calls nobody is waiting for. Batch APIs take a file of requests and return results later, at half price (as of October 2026):

Batch limits are separate from the real-time ones on both. On Anthropic, batch and caching discounts stack, but cache hits are "best-effort", typically "30 percent to 98 percent", and the docs suggest the 1-hour cache lifetime for shared context because batches can outlast 5 minutes.

%% title: The nightly tagging job on the Batch API
sequenceDiagram
  participant C as Nightly job (02:00)
  participant DB as Conversation store
  participant B as Batch API
  participant T as Tags table
  C->>DB: Yesterday's closed conversations (about 60,000)
  C->>B: Create batch, custom_id = conversation id
  B-->>C: batch id, in progress
  loop every few minutes
    C->>B: Poll status
  end
  B-->>C: Ended, results file
  C->>T: Upsert tags by custom_id (idempotent)
  Note over C,T: errored or expired rows join tomorrow's batch

custom_id is the idempotency key (Week 2, Day 2 · Idempotency): results come back in any order and a re-run must not double-write. Good candidates: tagging and summaries, Day 4's offline eval runs, backfills after a prompt change, and re-embedding (Day 2; OpenAI's batch endpoint also takes embeddings). Bad ones: anything a person is waiting for.

06Chapter 6: Budgets and a Budget Governor#

6.1 Provider Limits Aren't a Plan#

Day 1 covered the provider's limits, including monthly spend caps whose 429 has no retry-after. Anthropic also lets you set your own lower spend limit (requests then fail with a 400) and give each workspace its own spend and rate limits, a useful outer wall between features. But provider limits know nothing about tenants or users, and they fail by stopping everything at once. You want limits that know who is spending and degrade gradually.

6.2 Reserve, Then Settle#

A budget governor is a rate limiter whose unit is money, keyed by feature, tenant, user and conversation (Week 1, Day 3 · Rate Limiting has the counters). A call's cost is unknown until it ends, so reserve the worst case before the call and settle the real cost after. max_tokens bounds the worst case:

class BudgetGovernor:
    """Daily USD budgets for every key a call touches (feature, tenant, user, conversation)."""
    def __init__(self, store, limits: dict[str, float]):
        self.store, self.limits = store, limits          # store: atomic counters with a TTL

    def reserve(self, keys: list[str], worst_case: float) -> bool:
        taken = []
        for k in keys:
            if self.store.incrbyfloat(f"spend:{today()}:{k}", worst_case) > self.limits[k]:
                for t in taken + [k]:                     # roll back every key we touched
                    self.store.incrbyfloat(f"spend:{today()}:{t}", -worst_case)
                return False
            taken.append(k)
        return True

    def settle(self, keys: list[str], reserved: float, actual: float):
        for k in keys:                                    # actual = request_cost(model, usage)
            self.store.incrbyfloat(f"spend:{today()}:{k}", actual - reserved)

# worst case for one call: all input at the full input price + max_tokens at the output price

A sensible max_tokens keeps reservations honest; a huge one makes every reservation huge and starves the budget.

6.3 Degrade Before You Cut Off#

%% title: A spend ladder for one feature's daily budget (illustrative thresholds)
stateDiagram-v2
  [*] --> Normal
  Normal --> Thrifty: 80 percent of budget spent
  Thrifty --> Minimal: 95 percent spent
  Minimal --> Handoff: budget spent
  Thrifty --> Normal: new day
  Minimal --> Normal: new day
  Handoff --> Normal: new day
  note right of Thrifty
    small model for every turn
    lower effort, shorter replies
  end note
  note right of Minimal
    pause optional features
    move deferrable work to batch
  end note
  note right of Handoff
    polite message, human handoff
    never a raw error
  end note

The ladder is per feature: when the nightly tagging budget runs out, the chat keeps going. Alert at the first step, not the last.

07Chapter 7: Retries and Failover, Costed#

Day 1 set the retry rules for 429 and 529 (honour retry-after, jitter, a budget, never retry a spend-cap 429), and Day 4 built the fallback ladder. Two operational details remain.

Retry in one layer. The official SDK "automatically retries transient failures ... with exponential backoff, twice by default". Wrap it in your own 5-retry loop and a failing turn makes up to 6 × 3 = 18 calls; a retrying gateway in front multiplies again. Retries amplify load (Week 2, Day 3 · Circuit Breakers), so retry only in the gateway and build its clients with max_retries=0:

RETRYABLE = {429, 500, 504, 529}

def call_with_policy(req, providers, deadline: float):
    for p in providers:                                    # primary, then Day 4's tested fallbacks
        if p.breaker.is_open() or not p.ramp.admit():      # shift traffic a share at a time
            continue
        for attempt in range(3):
            try:
                return p.send(req)                         # SDK client built with max_retries=0
            except ProviderError as e:
                if e.code == "enforced_spend_limit_reached" or e.status not in RETRYABLE:
                    break                                  # retrying here can't succeed
                wait = e.retry_after or random.uniform(0, min(8.0, 0.5 * 2 ** attempt))
                if not retry_budget.take() or time.monotonic() + wait > deadline:
                    break
                time.sleep(wait)
        p.breaker.record_failure()
    raise AllProvidersFailed()                             # caller takes Day 4's handoff path

Failover has a price. The fallback provider has never seen your prefix, so its cache is cold: every input token is fresh, billed at the full input price and counted against the fallback's input-token limit. The fallback also runs a separately tested prompt variant (Day 4), with its own tokenizer and price list.

%% title: One turn during Thursday's 529s
sequenceDiagram
  participant A as Assist service
  participant G as LLM gateway
  participant P as Primary provider
  participant F as Fallback provider
  A->>G: Turn, deadline 8 s
  G->>P: Request
  P-->>G: 529 overloaded
  Note over G: Jittered backoff, retry budget OK
  G->>P: Retry 1
  P-->>G: 529 overloaded
  Note over G: Breaker opens on the 529 rate
  G->>F: Same turn, fallback prompt variant
  F-->>G: 200, cold cache, every input token at full price
  G-->>A: Reply, tagged provider = fallback

08Chapter 8: Observability and Cost Attribution#

8.1 One Span Per Model Call#

Traces, metrics and logs in general are Week 10, Day 2 · Observability. What's specific is what each model-call span carries. OpenTelemetry's GenAI semantic conventions (status: Development, so names may change) define it:

• Name: {gen_ai.operation.name} {gen_ai.request.model}, e.g. chat claude-sonnet-5-5.
• Required: gen_ai.operation.name, gen_ai.provider.name; gen_ai.request.model when available; error.type on failure.
• Recommended: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and when non-zero gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_write.input_tokens and gen_ai.usage.reasoning.output_tokens.
• Opt-in: prompt and reply text (gen_ai.input.messages, gen_ai.output.messages), because they hold customer data; Day 4's PII rules decide.
• Metrics: token counters such as gen_ai.client.inference.usage.input_tokens (input tokens "including cached tokens"), with cache and reasoning subsets; gen_ai.client.inference.time_to_first_chunk for streams.

One trap: OpenTelemetry's input count includes cached tokens, while Anthropic's input_tokens excludes them. The Anthropic mapping says: gen_ai.usage.input_tokens = input_tokens + cache_read_input_tokens + cache_write_input_tokens, where the last term is the response's cache_creation_input_tokens. Get it wrong and your token or cost totals are off by the size of your cache.

8.2 Who Spent It?#

Add your own attributes to every model-call span: feature (assist.chat, assist.router, guard.screen, guard.judge, qa.nightly_tags), tenant, prompt version, route (small, escalated, fallback) and cost in USD from the dated price table. Then Friday's question is a group-by. Day 4's screens and judges get their own feature names, so you can see what safety costs. Uber's GenAI Gateway (Day 1) does the same job with audit logs "for comprehensive cost attribution, security audit purposes, quality evaluation" (Uber Engineering, July 2024).

%% title: From one model call to a cost-per-feature chart
flowchart TD
  GW["LLM gateway<br/>one span per call: tokens, model,<br/>feature, tenant, prompt version, cost"]:::core
  P@{ icon: "sd:file", label: "Price table<br/>dated, per model" }
  C@{ icon: "sd:log", label: "OTel collector" }
  TS@{ icon: "tech:clickhouse", label: "ClickHouse<br/>spans and cost rows" }
  D@{ icon: "sd:dashboard", label: "Cost per feature, tenant,<br/>resolved conversation" }
  I@{ icon: "sd:file", label: "Provider invoice<br/>monthly" }
  AL@{ icon: "sd:alert", label: "Alerts<br/>cache-read share, spend rate, 529s" }
  P --> GW --> C --> TS --> D
  TS --> AL
  I -.->|"reconcile monthly"| TS
  class P,TS store
  class I external
  class C,AL,D service

Reconcile computed cost with the invoice monthly. A gap means a missing label, an unlogged path (batch jobs and retries are usual suspects) or a stale price.

The two lines that would have caught the preface early: cache-read share per prompt version (it fell to zero at 10:35) and spend rate per feature (the loop at 02:00).

09Chapter 9: Incident Patterns and Runbooks#

9.1 Provider Outage#

Providers go down completely. Between 3:16 PM and 7:38 PM PST on 11 December 2024, "all OpenAI services experienced significant degradation or complete unavailability". A new telemetry service made every node run Kubernetes API operations "whose cost scaled with the size of the cluster", overwhelming the control plane, and the fix needed that control plane, which they "could not do due to the increased load" (OpenAI status page write-up). For hours, retrying the primary helped nobody.

Runbook: breaker opens on the error rate; shift to the tested fallback in steps (Chapter 7); pause optional features and defer work to batch; tell users replies are slower; staff up the handoff queue.

9.2 The 429 Storm After a Deploy#

Runbook: check cache-read share by prompt version first (did the prefix change?), then retries as a share of calls; roll back the prompt; shed optional features down the spend ladder; ramp back gradually.

9.3 The Runaway Agent Loop#

Runbook: step caps per turn and budgets per conversation (Chapter 6); repetition detection; a kill switch per feature; fix the tool's error so it tells the model to stop.

9.4 The Quiet Quality Regression#

Day 4 gates prompt changes on evals. Regressions also come from below. Between August and early September 2025, "three infrastructure bugs intermittently degraded Claude's response quality"; one misrouted Sonnet 4 requests, from 0.8% at first to 16% in the worst hour on August 31, and because routing is "sticky", some users were hit far harder than the average. Anthropic wrote that its validation "ordinarily relies on benchmarks alongside safety evaluations and performance metrics", and responded with "more sensitive evaluations" and "quality evaluations in more places" (Anthropic, September 2025).

Runbook: prompt_version and gen_ai.response.model on every span; online quality signals (handoffs, thumbs-down, repeat contacts) per version and per model; sampled online judging (Day 4); one-click prompt rollback (Week 10, Day 3 · Deployment Strategies); a support path to report provider-side degradation with request IDs.

9.5 The Cache Key That Forgot the User#

Someone adds an answer cache and leaves the user out of the key; customer B sees customer A's refund. Runbook: turn the cache off by flag, purge it, find affected responses through the spans' cache-hit attribute, and run it as a privacy incident (Week 10, Day 5 · Incident Management). Prevention: one key-building function that refuses to run without a scope, with tests.

10Chapter 10: The Monthly Cost Model#

Every lever, applied to Day 1's MegaMart numbers.

11Chapter 11: The Whole Design#

%% title: MegaMart Assist, run economically (illustrative design)
flowchart TD
  U@{ icon: "sd:users", label: "Customers<br/>web and app chat" }
  AS["Assist service<br/>conversation state, tools, RAG (Day 3), guardrails (Day 4)"]:::core
  GW["LLM gateway<br/>budget governor · answer cache ·<br/>router · one retry layer · breaker"]:::core
  R@{ icon: "tech:redis", label: "Redis<br/>spend counters, answer cache" }
  S@{ icon: "sd:ml", label: "Small model<br/>Haiku 5.5" }
  L@{ icon: "sd:ml", label: "Large model<br/>Sonnet 5.5" }
  F@{ icon: "sd:cloud", label: "Fallback provider<br/>tested prompt variant (Day 4)" }
  N@{ icon: "sd:cron", label: "Nightly QA tagging<br/>Batch API, 50% price" }
  O@{ icon: "sd:log", label: "OTel spans<br/>tokens, cost, feature, prompt version" }
  D@{ icon: "sd:dashboard", label: "Cost and quality dashboards,<br/>alerts" }
  U --> AS --> GW
  GW <--> R
  GW --> S
  GW --> L
  GW -.->|"breaker open, stepped shift"| F
  N --> GW
  GW --> O --> D
  class U client
  class R cache
  class S,L,F external
  class N,O,D service

The gateway sees every call, so budgets, caches, routing, retries and telemetry live there. No service calls a provider directly.

12Chapter 12: Common Mistakes#

12.1 Volatile Content Above the Breakpoint#

Timestamps, names, request IDs or a reordered tool list early in the prompt make every prefix unique. Fix: stable first, volatile in the newest turn; alert on cache-read share per prompt version.

12.2 Counting Tokens Without the Cache Fields#

Charting only Anthropic's input_tokens hides most input once caching works, and makes cost and capacity charts lie. Fix: log all four meters and map them to OpenTelemetry's totals.

12.3 A Semantic Cache Without Scope or Measurement#

Keyed only by the question's meaning, it returns personal, stale or flipped answers with full confidence. Fix: scoped keys, non-personal intents only, invalidation from sources, a sampled false-hit rate.

12.4 One Budget for Everything#

One organisation-wide cap lets a runaway loop or a batch job take down the customer-facing assistant. Fix: budgets per feature, tenant, user and conversation, reserved before the call, with a ladder that degrades before it stops.

13Chapter 13: Interview Section#

13.1 When to Bring It Up#

Any design with an LLM in the request path. Bring up cost as soon as you've estimated tokens, because it shapes the architecture: prompt order, model choice, what runs in batch, where the gateway sits.

13.2 Phrases That Work#

13.3 Follow-Up Questions#

14Chapter 14: Practice Problems#

Summary#

Further Reading#

Official Documentation#

• Docs

  Anthropic — Prompt caching: https://platform.claude.com/docs/en/build-with-claude/prompt-caching

• Docs

  Anthropic — Pricing: https://platform.claude.com/docs/en/about-claude/pricing

• Docs

  Anthropic — Batch processing: https://platform.claude.com/docs/en/build-with-claude/batch-processing

• Docs

  Anthropic — Rate limits: https://platform.claude.com/docs/en/api/rate-limits

• Docs

  Anthropic — Errors: https://platform.claude.com/docs/en/api/errors

• Docs

  Anthropic — Effort: https://platform.claude.com/docs/en/build-with-claude/effort

• Docs

  OpenAI — Prompt caching: https://developers.openai.com/api/docs/guides/prompt-caching

• Docs

  OpenAI — Batch API: https://developers.openai.com/api/docs/guides/batch

• Docs

  Google — Gemini API context caching: https://ai.google.dev/gemini-api/docs/caching

• Read

  OpenTelemetry — GenAI semantic conventions: https://github.com/open-telemetry/semantic-conventions-genai

• Read

  Cloudflare — AI Gateway caching: https://developers.cloudflare.com/ai-gateway/features/caching/

Papers and Libraries#

• Paper

  Chen, Zaharia and Zou — FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023): https://arxiv.org/abs/2305.05176

• Paper

  Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data (2024): https://arxiv.org/abs/2406.18665

• Read

  GPTCache — A Library for Creating Semantic Cache for LLM Queries: https://github.com/zilliztech/GPTCache

Engineering Blogs and Incident Reports#

• Blog

  Uber — GenAI Gateway (July 2024): https://www.uber.com/blog/genai-gateway/

• Docs

  OpenAI — API, ChatGPT and Sora facing issues: write-up (December 2024): https://status.openai.com/incidents/ctrsv3lwd797/write-up

• Blog

  Anthropic — A postmortem of three recent issues (September 2025): https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues