Your circuit breaker cut off every payment on Black Friday

transcript

Black Friday. Your busiest day. Every payment gets cut off. The bank is perfectly healthy. Your circuit breaker did this.

The breaker's rule: 5 failures, and it opens. Counted within 60 seconds. And the bank? A tiny, normal 0 point 05 percent of calls time out. Every day.

A normal day: 100 requests a second. About 3 timeouts a minute. Never trips. Black Friday: 10,000 a second. Same 0 point 05 percent. That's now 5 timeouts every second. The breaker opens in the first second. Every payment cut off.

The bank isn't failing more. You just have more traffic. 5 out of 100 is worrying. 5 out of 10,000 is excellent. A count can't tell the difference.

So open on a rate, not a count. Say, half of calls failing. And only after at least 100 calls. 1 failure in 2 is 50 percent, and means nothing. And a declined card? That's the bank working, saying no. Count it as a success.

A fixed count punishes success. More traffic, more normal failures. Open on a rate. Then traffic alone can never trip it.

Full breakdown in the lecture.