Turning an AI assistant's tools off mid-incident can trigger a 429 storm

transcript

Refunds are down mid-evening. Your AI assistant must stop taking actions. Easy fix, right? Just remove the tools from the request. Done.

Here's the catch. The prompt cache only matches an exact prefix. Same bytes, same order. And tool definitions are the very first thing in it. Change them, and everything after them is new.

With the cache, about 290,000 input tokens a minute count against the limit. Miss the cache, and it's about 3.45 million. Mid-incident. At the evening peak. The provider answers 429. Too many tokens.

Removing tools turns every request into a cache miss. Exactly when you're weakest. A second incident, on top of the first. Instead, keep the tools byte for byte. The gate says no. With an error the model can use. Refunds are paused. Offer an agent.

Don't edit the prompt to change behaviour mid-incident. Tool definitions stay identical. Byte for byte. Keep the prefix. Let the gate say no.

Full breakdown in the lecture.