At 09:19 Pacific time on 13 August 2015, lightning struck the local power grid that feeds Google's data center in St. Ghislain, Belgium, the zone Google Cloud calls europe-west1-b. It struck four times in succession.
Automatic backup systems restored power quickly, and the storage systems were designed with battery backup. But some data that had just been written was sitting on storage systems that were more vulnerable to power loss when the batteries were drained for longer, or repeatedly. That is where the damage happened.
The affected product was Persistent Disk, the virtual hard drives that customers attach to their cloud servers. About 5% of the standard persistent disks in the zone returned at least one read or write error during the incident. Some management operations, like taking a snapshot of a disk, failed too. SSD disks, local SSDs and existing snapshots were not affected.
Google engineers spent the next four days recovering data, and the number of affected disks kept falling. In almost every case, the data was safely saved in the end. But a very few recent writes could not be recovered: less than 0.000001% of the allocated disk space in the zone was permanently lost.
Google took full responsibility and said it was upgrading to storage hardware that holds cached data better through power loss. It also used its report to remind customers of a basic fact: a zone is a single data center, so anything kept in only one zone can be lost in a data-center-scale disaster. For durability it pointed to snapshots and Cloud Storage, which are copied across locations.
- 4 lightning strikes in succession
- About 5% of standard persistent disks in the zone saw errors
- 13 to 17 August: four days of recovery
- Less than 0.000001% of disk space permanently lost
Why it matters: "Rare" is not "never". If data matters, keep copies in more than one place, so that one building's bad day can't destroy it.