On 31 January 2017, spam and a background job (deleting an employee's account wrongly flagged for abuse) were loading GitLab.com's database. The extra load made the secondary database, a live copy kept in sync with the primary, fall behind. Then replication broke: the primary had already thrown away the change logs the secondary needed to catch up.
An engineer set out to rebuild the secondary from scratch. The first step was to wipe its data directory. At around 23:30 UTC they ran the command, "errantly thinking" they were on the secondary. It ran on the primary, the real database. They stopped it within seconds, but about 300 GB of data was already gone.
Then the team went looking for backups.
- The regular database dumps meant to go to Amazon S3: the bucket was empty. The backup job used PostgreSQL 9.2 tools against a 9.6 database, failed silently, and its failure emails were being rejected.
- Azure disk snapshots: not enabled for the database servers.
- The secondary: its data had just been wiped as part of the rebuild.
What saved them was luck. About six hours earlier, an engineer had manually taken a disk snapshot to get fresh data for load testing. GitLab restored from that. Copying it back took about 18 hours on slow cloud disks. Everything written in those six hours was lost: roughly 5,000 projects, 5,000 comments and 700 new user accounts.
GitLab handled it in the open. It kept a public Google Doc of its progress and livestreamed the recovery on YouTube, with around 5,000 people watching at the peak. Its postmortem focused on better disaster recovery, not on stopping engineers from running commands.
- About 300 GB deleted in seconds
- 6 hours of data lost; about 18 hours to restore
- About 5,000 people watching the livestream at peak
Why it matters: A backup you've never restored is a hope, not a backup. Test restores regularly, alert when backup jobs fail, and make "which machine am I on?" impossible to get wrong.