Multiple copies, one bug: the night tape saved Gmail

transcript

In 2011, Gmail kept multiple copies of your mail, in multiple data centres. One software update wiped several of them at once.

February 2011. A storage software update rolls out. It has a bug. 0.02 percent of Gmail users find empty accounts. Google stops the rollout. Reverts. Then it restores the mail. From tape.

But wait. Why didn't the other copies save it? Replicas save you when one disk dies. The others survive. A bug isn't one disk. It runs on every replica. And a delete replicates faithfully.

The tapes were offline. The bug couldn't touch them. Restoring took hours, not milliseconds. But the mail came back.

So keep one copy your software can't reach. Make deletes slow. Purge only after a grace period, so a mass delete can be undone. Roll out storage code one cell at a time. Canary, 1 percent, 10.

Replicas fail independently. Bugs don't. Keep one copy your own code can't reach.

Full breakdown in the lecture.