Errors spike after a deploy. Don't hunt for the bug yet.

transcript

Errors are spiking after a deploy. Your instinct: find the bug. Wrong first move. Stop the bleeding first.

First question: was there a recent deploy? Can you roll back safely? No? Is a dependency down? Fail over, or degrade gracefully. No? Are you overloaded? Scale up, or rate limit.

The lesson's incident. A deploy at 14:30. Errors start rising at 14:35. The alert fires at 14:42. 7 minutes to detect. On-call links it to the deploy. Rollback at 14:55. 15:00: errors back to normal. Mitigated in 18 minutes. Nobody knows the bug yet.

The SEV 1 targets. Detect in under 5 minutes. Engage in under 15. Mitigate in under an hour.

The root cause is resolution. It can take longer than mitigation. And that's OK. Here it was N plus 1 queries, found by the postmortem. After users were fine.

Restore service with the fastest safe move, even if it isn't the real fix. Users first. Root cause second.

Full breakdown in the lecture.