A slow service is worse than a dead one

transcript

A dependency of yours goes down. Annoying. You handle it. Now it just gets slow instead. That's the one that takes you down with it.

Your server has 100 threads. Each request makes one downstream call, normally 100 milliseconds. So each thread does 10 requests a second. Capacity: 1,000. Bad day. The downstream takes 10 seconds. Each thread manages a tenth of a request a second. Capacity: 10 a second. A hundred times less. And your code didn't change at all.

Down is honest. You get an error straight away. Plan B. Your threads stay free. Slow is a trap. You don't know if it'll ever answer. So every thread just waits. New requests queue. The queue fills. You reject everything. To your callers, you look dead.

And it climbs. A calls B, B calls C. C slows to 10 seconds. One second in, half of B's threads are stuck. Five seconds in, all of them. Second 6, A's threads are waiting on B. Second 10, the user sees a timeout. They only called A. They don't even know C exists.

So what's a timeout actually for? It turns slow into down. Give up, free the thread, and handle it like the honest failure it is.

A dead service gives you an answer. A slow one takes your threads. Slow is worse than down. So decide how long you'll wait, before it happens.

Full breakdown in the lecture.