Your monitoring dies at 3 a.m. Who tells you?

transcript

Your monitoring system just died. Who's going to page you? Not the monitoring system. It's dead.

Here's the trap. A dead monitor sends no alerts. And no alerts looks exactly like a quiet night.

This system has one rule. Dashboards may degrade. Alerting must not. So alerts read their own consumer. Rules run every 30 seconds, on fresh data. A slow dashboard query can never delay a page.

But that still can't catch its own death. So add a watchdog. An alert that fires all the time. On purpose. Every minute, it goes to a service outside the system. If it stops arriving, the outside service pages you.

A normal night. The watchdog arrives every minute. No pages. The worst night. The watchdog stops. A page from outside. Your monitoring system is down. And now you know.

A failed system can't report its own failure. So something outside has to notice the silence.

Full breakdown in the lecture.