Dropbox moved 500 petabytes off Amazon in a sprint called BASE Jump

transcript

Dropbox gave itself six months. Grow its own storage past 500 petabytes. Its users' files, out of Amazon's cloud. Without losing any?

Dropbox already ran its metadata and web servers. But the files themselves? On Amazon S3. 2012: about 40 petabytes of user data. 2016: 500 petabytes. More than 500 million sign-ups.

Summer 2013. A small team starts building their own storage. Magic Pocket. August 2014. The dark launch. Mirrored across two regions, ready for user data. But they kept extra backups for six more months. Just in case. 27 February 2015. For the first time ever, files are served only in-house.

Then the aggressive goal. Past 500 petabytes, in six months. They named the sprint BASE Jump. No time to open the parachute.

From 30 April 2015, they installed servers in three regions. Racing the incoming data. The network moved over half a terabit per second, at peak. Goal: 90 percent in-house by 30 October. They hit it on 7 October.

No major disruptions. No lost data. And their IPO filing? Infrastructure costs fell 39 point 5 million dollars in 2016. Then another 35 point 1 million in 2017.

Migrate in stages. Build it, run it dark, keep backups after the switch. Then scale. At steady scale, owning can cost far less than renting.

Full breakdown in the lecture.