40 model calls in one conversation, and nothing said stop

transcript

2 in the morning. One customer. 40 model calls. And nothing told it to stop.

An order-lookup tool keeps failing. The model reads the error. And calls the tool again. Every call resends the whole history. And it keeps growing.

Will the provider's spend cap stop it? Eventually. By stopping every other customer too.

The fix is layers. First, a step cap per turn. Say 5 tool calls. Then hand off to a person. Next, a budget per conversation. It catches loops you didn't predict. Feature budgets and the provider cap come much later.

A budget is a rate limiter. Its unit is money. Before each call, reserve the worst case. Max tokens bounds it. After the call, settle the real cost from usage.

At 80 percent of the day's budget: small model, shorter replies. At 95: pause optional features. Defer work to batch. Budget spent? A polite handoff to a human. Never a raw error.

Budgets shouldn't fail like a wall. Cap the loop. Degrade first. Stop last.

Full breakdown in the lecture.