The circuit breaker worked. The outage went unnoticed for 75 minutes.

transcript

Your error rate drops. Latency looks great. And nobody can log in. For over an hour.

10 AM. A bad deploy breaks the authentication service. A minute later, the circuit opens. And the error rate drops. Nothing is being called. The on-call engineer sees circuit open, but low errors. Maybe transient. 10:30. Average latency looks great. Most calls fail in milliseconds.

11 AM. A customer complains they can't log in. 11:20, the deploy is found and rolled back. Time to detect: 1 hour 15 minutes. Total outage: 1 hour 25 minutes.

Here's the twist. The errors weren't fixed. They were prevented. Calls never happened. Latency was great because failing fast is fast. And one failed probe every 30 seconds looks like flakiness. Not an outage.

So alert on the circuit, not just the errors. Export its state: closed, half open, open. Open for over a minute? Critical. Investigate the downstream. Flipping state over 5 times in 10 minutes? A warning. Something's unstable.

A breaker protects you by stopping calls. That also stops the errors you alert on. Even when it's working perfectly, an open circuit is worth investigating.

Full breakdown in the lecture.