The 20:00 retry storm: 3.45M input tokens a minute against a 2M limit, retried at once

transcript

20:00. The sale starts. Your AI support assistant is ready for it. Requests are under the provider's limit. Rate-limit errors start anyway.

The real limit is tokens. It needs about 3.45 million input tokens a minute. The limit is 2 million. Every turn re-sends a long prompt and the history.

Too many tokens. The provider answers 429. Every app server retries at once. Up to three times. Same demand. Up to four times the attempts. Which all fail again. A retry storm.

But here's the catch. Retrying didn't add capacity. It just multiplied the load on a limit you'd already hit.

Instead, put a gateway in front. One shared token bucket. Don't send what will be rejected. Honour retry-after, with jitter. Live chat goes first. Batch work waits its turn, or moves to a batch API.

Then fix the cause. Trim the history. Cache the shared prompt. And raise the tier before the sale. Not during it.

Full breakdown in the lecture.