Multi-window burn-rate alerts.
Turn per-minute good and bad counts into error ratios and burn rates, page only when a long and a short window both burn fast, and report the budget left.
"Alert if the error rate is above 0.1%" fires on every blip and says nothing about urgency. Day 1 replaces it with burn-rate alerting (section 4.3): how fast are we spending the month's error budget?
budget = 1 - slo # 0.001 for a 99.9% SLO
error_ratio = bad / total # over a window
burn_rate = error_ratio / budget # 1.0 spends the budget in exactly one SLO periodA multi-window rule fires only when both a long window and a short window burn faster than its threshold. The long window proves the problem is big enough to matter; the short window proves it is still happening, so the page stops soon after the fix instead of an hour later. The given RULES are the lesson's four rows, from 14.4× over 1 hour (with a 5-minute short window) down to 1× over 3 days.
Implement BurnRateMonitor(slo, rules=RULES, period_minutes=43200):
| Method | Returns |
|---|---|
record(minute, good, bad) |
nothing. Adds the counts for that minute (an int). The same minute can be recorded more than once, in any order |
error_ratio(now, window) |
bad / total over the window minutes ending at now: minutes now - window + 1 to now, both included. 0.0 with no traffic |
burn_rate(now, window) |
error_ratio / (1 - slo) |
firing(now) |
the names of the rules firing at now, in rules order. A rule fires when the burn rate over its long_window and over its short_window are both above (>) its burn_rate |
budget_remaining(now) |
the fraction of the error budget left over the last period_minutes: 1 - bad / (budget × total). 1.0 with no traffic; below 0 once overspent |
Rules the tests check:
- Ratios are traffic-weighted: add up the bad and total counts over the window, then divide. Don't average per-minute ratios: one minute with 10 failed requests out of 10 isn't 50% of a window that also served 1,000 good ones.
- Minutes after
nowdon't count (the tests replay history and ask about the past). - A window with no traffic burns at
0.0, it doesn't divide by zero.
A 30-day budget at burn rate 1 lasts 30 days. At 14.4× it lasts 30 × 24 / 14.4 = 50 hours, and one hour at that rate spends 1/50 = 2% of the month's budget: the lesson's "2% in 1 hour". Use math.isclose thinking here: 1 - 0.999 is 0.0010000000000000009 in floating point, so the tests stay clear of exact thresholds.
Write your code, then run 3 example tests and 9 hidden tests. Each run is limited to 8 seconds: an infinite loop gets stopped and Python restarts, your code stays.