900 requests a minute, and still rate-limited: AI APIs count tokens

transcript

900 requests a minute. At your busiest hour. Well under your limit. And your AI assistant still gets rate-limited. How?

Your database counts requests. A model provider counts more than that. It also counts input tokens a minute. Every word you send. And output tokens a minute. Every word it writes back.

But here's the catch. The model remembers nothing between turns. So every turn, your app re-sends the instructions. And the whole conversation so far.

First, a long prompt of instructions. On every single turn. Then the history. It grows with every turn of the chat. Then the customer's new message. The tiny part at the end. One request on the dashboard. A pile of tokens on the bill.

Requests? 900 a minute. Comfortably under. Tokens? Prompt plus history, on every turn. Tokens hit the limit long before requests do. That's the surprise.

So stop sizing an AI feature by requests. Size it in tokens a minute. In, and out.

Full breakdown in the lecture.