GitLab wiped its main database. Then it looked for the backups.

transcript

One command. 300 gigabytes. Gone. And it wasn't the copy. It was GitLab's real database.

31 January 2017. Spam and a background job are hammering GitLab's database. The live copy falls behind. Then replication breaks. So an engineer rebuilds the copy. Step one: wipe it. They run the command. But they're on the primary. They stop it in seconds. Too late.

No panic yet. They have backups. Watch what happens. Database dumps to Amazon S3. They open the bucket. Empty. Azure disk snapshots? Not enabled for the database servers. The live copy? They'd just wiped it, for the rebuild.

But here's the twist. Something did save them. Six hours earlier, an engineer took a disk snapshot. Just for load testing.

They restored from it. Six hours of data, gone. Roughly 5,000 projects. 5,000 comments. 700 new accounts. And copying it back? About 18 hours, on slow cloud disks.

Why was the bucket empty? The job ran Postgres 9 point 2 tools. On a 9 point 6 database. So it failed. Silently. The bucket stayed empty. And the failure emails? Rejected. Nobody got the warning.

GitLab's fix wasn't stopping engineers. It was better recovery. Test your restores. Alert on failed backups. Always know which machine you're on. Because a backup you've never restored is a hope. Not a backup.

Full breakdown in the lecture.