A bug emptied Gmail inboxes. Tape brought them back.

transcript

Imagine logging in to Gmail. And your inbox is empty. All your mail. Gone? An old-school backup got it back.

27 February 2011. It happens to 0 point 0 2 percent of Gmail users. But Google keeps several copies of your mail. In several data centres.

Here's the villain. A storage software update, with a bug. It didn't break a disk. It ran on the copies. And deleted mail. More copies don't help when the same bug reaches them.

But here's the twist. Google also backed up to tape. Tapes sit offline. No software bug can touch them. The catch? Hours to restore. Not milliseconds.

Google stops the rollout. Reverts. Then starts reading tapes. 30 hours of work, and counting. 1 March. The last 0 point 0 1 2 percent, restored from tape. 2 March. Gmail back to normal for nearly everyone.

Old mail came back from tape. But mail sent to them over about 20 hours likely never arrived. Senders got notices.

Replicas save you from broken machines. Not from broken software. Keep one backup offline. Where your code can't reach it.

Full story in the article.