Lightning struck a Google data center's grid four times in a row

transcript

Lightning hit the grid feeding a Google data center. Four times. Power came back fast. Some data still vanished forever.

09:19 Pacific. St. Ghislain, Belgium. Lightning strikes the local grid. Again. And again. Backup systems restore power quickly. And the storage has batteries. Yet about 5 percent of standard persistent disks start throwing errors. Engineers spend the next four days recovering data.

So if power came back, why did anything die? Batteries cover one short outage. But this wasn't one outage. Four strikes drained them for longer. Or again and again. Data just written sat on storage more vulnerable to exactly that. A very few recent writes never came back.

Four days of recovery. The affected disks kept falling. In almost every case, the data was saved. Lost forever: under a millionth of a percent of disk space. Tiny. Unless it's yours.

Google took full responsibility. Then it reminded customers of a basic fact. A zone is a single data center. Keep data in one, and one disaster can take it. For durability, use snapshots and Cloud Storage. They're copied across locations.

Four strikes in a row sounds rare. But rare is not never. If data matters, keep copies in more than one place.

Full breakdown in the lecture.