Half the chats in Hindi on a sale evening: output, not input, hits the limit

transcript

8 pm. A festival sale. Half your support chats are in Hindi. Your AI assistant's token limits were planned for a normal evening. So what breaks first?

Traffic doubles. About 1,800 turns a minute. Hindi takes about three times the tokens of the same English. So a 250-token reply becomes about 750 tokens.

The plan said 225,000 output tokens a minute. About half the limit. Double the traffic: 450,000. Half in Hindi: 900,000. Four times the plan.

But you cached your prompts! Doesn't matter tonight. Caching only discounts input. It can't shrink what the model writes. And each Hindi reply takes about 15 seconds to finish.

So before the sale starts, raise the output limit. Ramp up, don't jump. Ask for shorter replies, with a token cap per language. And put Hindi-speaking agents on shift for handoffs.

Same chats. Different language. A different limit binds. Caching shrinks input. On a sale evening, output binds first.

Full breakdown in the lecture.