One update, every machine at once: the canary CrowdStrike didn't have

transcript

One update. Sent to every eligible machine at once. About 8 point 5 million Windows machines crashed.

CrowdStrike updates its Falcon sensor two ways. The slow way is new code. That gets release testing, then Microsoft's driver certification. The fast way is channel files. Straight from the cloud. No code change. On 19 July 2024, Channel File 291 went to every eligible online sensor.

Here's the question to ask. If this change is bad, how much breaks? For this file? Potentially every online Windows machine on recent Falcon.

The fix is a staged rollout. First, a small canary ring. Then you wait. That's the bake time. Any crashes? This bug crashed machines fast. The canary would have lit up fast. Roll back. Healthy? Widen ring by ring, until every sensor has it.

Now the maths, with made-up ring sizes. No rings: about 8 point 5 million devices. A first ring of 1 percent: about 85,000. A hundred times fewer. A canary of 5,000, halted at the first crash? Just 5,000.

You keep the speed. And you gain a brake. Pushing a config change everywhere at once? Picture the canary that wasn't there. Every fast path needs a brake.

Full breakdown in the lecture.