Errors hit 5% for 3 minutes. Nobody got paged. Correctly.

transcript

Errors jump to 5 percent. Then fully recover. Nobody gets paged. And that's exactly right.

The old alert fires when errors pass 0 point 1 percent. Every tiny spike. A burn rate alert asks: how fast is the budget draining?

Burn rate is the actual error rate, divided by the allowed one. Errors at twice the allowed rate? A burn rate of 2.

1 times, over 3 days? Review it in the weekly meeting. 3 times over a day: a ticket. 6 times over 6 hours: page in business hours. 14 point 4 times over an hour: page now. 2 percent of the month's budget, gone in an hour.

Now the blip. Errors hit 5 percent for 3 minutes, then fully recover. Over a 5 minute window, the burn rate is far above 14 point 4. Over the hour, it averages out to 0 point 2 5 percent. Roughly 2 point 5 times. The rule needs both windows over 14 point 4. So nobody wakes up.

Short blips that barely dent the budget don't wake anyone. Real, sustained burns do.

Full breakdown in the lecture.