On 27 February 2011, some Gmail users logged in and found their accounts empty. Google's own post opened with exactly that: "Imagine the sinking feeling of logging in to your Gmail account and finding it empty." It happened to 0.02% of Gmail users.
Google kept multiple copies of everyone's mail, in multiple data centres. But this wasn't a hardware failure. Google had released a storage software update with a bug, and, as the post put it, "in some rare instances software bugs can affect several copies of the data." Copies of mail were deleted. Having more copies didn't help when the same bug could reach them.
When Google found the problem, it stopped rolling out the update and went back to the old version. Then it reached for the copy the bug couldn't reach: tape. Google also backed mail up to tapes, and because tapes sit offline, software bugs can't touch them. The catch is speed: restoring from tape took hours, instead of the milliseconds it takes to send your requests to another data centre.
By the first post, engineers had been at it for 30 hours. On 1 March, data for the remaining 0.012% of affected users had been restored from tape, and Google planned to start moving it back into mailboxes within hours, bigger accounts taking longer. Late on 2 March, Google said Gmail should be back to normal for the vast majority of affected people.
One thing tape couldn't fix: affected users likely didn't receive mail sent to them between 6 p.m. PST on 27 February and 2 p.m. PST on 28 February. The senders got notices that it wasn't delivered.
- 0.02% of Gmail users found their mailboxes empty
- 30 hours of recovery work by Google's first post
- 0.012%: the last affected users, restored from tape by 1 March
- About 20 hours of incoming mail not delivered to affected users
Why it matters: Replicas protect you from broken machines, not from broken software: a bug runs on every copy. Keep at least one backup offline, and roll out storage changes slowly.