CrowdStrike fixed the file in 78 minutes. The machines took days.

transcript

CrowdStrike fixed its bad file in 78 minutes. So why did the world spend days fixing machines?

A machine crashes and restarts. But the bad file is still on disk. Falcon's driver loads early in boot. It reads the same file. Blue screen. Restart. Crash again. A boot loop. The fixed file sits in the cloud. The machine never stays up to fetch it.

The early advice? Reboot, and hope it stays up long enough. Some Azure machines needed as many as 15 reboots. Many never won the race.

So someone sits at the keyboard. Boot into Safe Mode. The driver stays off. Delete the files starting C 00000291. Restart. One machine done. Now do thousands. But here's the catch. Encrypted disk? First type a 48-digit BitLocker key. Often kept in a key system that may itself run on Windows.

Delta's chief executive said they reset 40,000 servers by hand. Delta cancelled more than 5,000 flights. His figure for the cost? About 500 million dollars, over five days.

If an update stops a machine booting, your update channel can't fix it. Know where your recovery keys are. Practise the hands-on fix. Fixed in 78 minutes. Days to fix the machines.

Full breakdown in the lecture.