Dropbox's OS upgrade script reinstalled machines that were still serving traffic.

Dropbox ran on thousands of databases. Each had one master and two replicas for redundancy, and there were full and incremental backups stored in a separate environment.

On Friday 10 January 2014, at 5:30 p.m. PT, Dropbox began planned maintenance to upgrade the operating system on some of its machines. The upgrade script had a safety check built in: before installing the new OS, it made sure there was no active data on the machine.

The script had a subtle bug. It reinstalled a small number of active machines, ones still serving production traffic. Among them were master-replica pairs: a database's master and its copy, hit together. The site went down.

Files themselves were never at risk: these databases didn't hold file data. They powered features like photo album sharing, camera uploads and some API features.

With the replicas gone, the only way back was the backups. Dropbox restored most functionality within about 3 hours. But some databases were large, and the standard tool for recovering MySQL from backups is slow with big data sets. Core service didn't fully return until 4:40 p.m. PT on Sunday, almost two days after the maintenance began.

Dropbox drew two lessons. First, a remote check isn't enough: machines now also verify their own state before running incoming commands, so a machine running critical processes can refuse a destructive one. Second, restores must be fast: it built a tool that replays MySQL's binary logs in parallel, to recover large backups much faster.

Why it matters: Replicas don't help when one command reaches the master and its copies. Let critical machines refuse destructive commands themselves, and make sure you can restore your biggest database quickly.