A circuit breaker opened at 3 AM. Should anyone wake up?

transcript

3 AM. The circuit breaker for SendGrid opens. Should someone get woken up? Probably not.

An email needs sending. SendGrid's circuit is open, so it's skipped. The provider manager takes the next one. SES. The user gets their email. They never knew.

So page on symptoms. Delivery success under 95 percent, for 5 minutes. Every provider for a channel down, for 2 minutes. Lag over a million, for 10. Causes only warn. One open circuit, for 5 minutes. A slow provider.

Notice every rule says for. A dip that recovers by itself isn't actionable. Brief blips page nobody. Only sustained drops do.

And when a circuit does stay open, check each provider its own way. SendGrid: its status page, the API key, your IP reputation. Twilio: the account balance. It stops sending at zero, even with a green status page.

Paging on every open circuit just causes alert fatigue. Page on what users feel. Warn on what might be causing it.

Full breakdown in the lecture.