One CrowdStrike content update crashed an estimated 8.5 million Windows computers

transcript

An estimated 8 point 5 million Windows computers. Blue screens. Because of one field that didn't exist.

CrowdStrike's sensor runs deep inside Windows. It gets new detection rules from the cloud. 19 July 2024. 04:09 UTC. A new rule file ships. Machines that load it crash. Blue screen. 05:27. CrowdStrike pulls the file. 78 minutes after it shipped.

The villain was born in February. A new rule type for sensor 7 point 11. Its definition says it takes 21 input fields. The sensor code feeding it supplies 20.

But here's the catch. Nothing broke for months. Every test, and four rule sets since March, used a wildcard there. Match anything.

July's new rule used a real value in field 21. The validator checked it against 21 fields. Passed. The sensor reached for input 21. Past the end of a 20-item list. Windows crashed.

The file was live for 78 minutes. Microsoft estimated 8 point 5 million Windows devices. Less than one percent of them all. Macs and Linux machines? Untouched.

The fixes. Check the field count when the sensor is built. Check bounds on every read. And roll rules out in stages. Canary machines first, then wider rings.

If it can crash a machine, it's code. Ship it like code. In stages. Never all at once.

Full story in the article.