The timeout that tuned itself, and hid a bug for 3 hours

transcript

Your timeout was smart. It tuned itself. That's exactly how a bug hid for 3 hours.

The idea is lovely. Set the timeout to the recent P99, times 3. Hour 1: P99 100, timeout 300. Hour 2, the service degrades to 500. The timeout follows, to 1,500. Hour 3, it recovers. Timeout drops to 240. No manual tuning, ever.

Now a bug makes the service slow. With a fixed 500 millisecond timeout, requests start timing out. An alert fires. The team digs in. Bug found in 30 minutes. With the adaptive one? It quietly climbs to 2,000 milliseconds. Requests succeed, slowly. No timeouts, so no alert. Users complain. Bug found in 3 hours.

Here's the twist. The timeout didn't fail. It adapted. To the bug. It traded a loud failure for a quiet one.

So alert on the timeout value itself. Above 2,000 for 5 minutes? Raise a warning. Give it a hard maximum it can never pass. And use it for external APIs and variable work. Not where slow means a bug.

A timeout isn't just a limit. It's an alarm. Don't let it quietly switch itself off.

Full breakdown in the lecture.