50% more traffic didn't make the API 50% slower. It started a spiral.

transcript

Traffic grows by half. Your API doesn't get half as slow. It spirals into failure. One equation says why.

Little's Law: requests in flight equal arrival rate times time. 100 requests a second. Each takes 200 milliseconds. 100 times 0 point 2: 20 requests in flight at any moment.

You run 10 pods with 2 workers each. 20 workers. Exactly at capacity. Now traffic grows.

150 a second. Now you need 30 in flight. You only have 20 workers. So requests queue. Time in the queue adds to each request's time. W grows. Bigger W means bigger L. More queueing. Bigger W again.

Now plan with it. 500 a second at 200 milliseconds: 100 in flight. At 4 workers a pod, 25 pods. Run at 70 percent for headroom: about 36 pods. That's your plan.

Know your arrival rate and your latency. Then you know your workers. Run at capacity, and a small rise becomes a spiral.

Full breakdown in the lecture.