Dropbox's upgrade script had one job: skip live machines.

transcript

The script had one safety check. Skip any machine with live data. It missed. Core service took almost two days to fully come back.

Dropbox ran thousands of databases. Each one: a master and two replicas. Plus backups, kept somewhere else. Safe, right?

Friday, 10 January 2014. 5:30 p.m. Planned OS upgrades begin. Before installing the new OS, it checks. Any live data here? A subtle bug. It reinstalls a few machines still serving traffic. Some were a master and its replica. Together. The site goes down.

Your files were never at risk. These databases didn't hold them. They ran features. Photo albums, camera uploads, parts of the API.

With the replicas hit, only one way back. The backups. Most features came back in about three hours. But here's the catch. Big MySQL restores are slow. Core service fully back? Sunday, 4:40 p.m. About 47 hours.

Fix one. Machines now check themselves before obeying a command. Running something critical? Refuse the destructive command. Fix two. A tool that replays MySQL logs in parallel. Much faster restores.

Replicas can't save you when one script hits them together. Let critical machines say no. And practise restoring your biggest database.

Full story in the article.