One command took Facebook off the internet, and locked its engineers out

transcript

This one command took Facebook off the internet. For nearly six hours. And it locked out the engineers trying to fix it.

Facebook's data centers are linked by its own fiber backbone. 4 October 2021. Routine maintenance. A command checks available capacity. Instead, it takes down every backbone connection. A safety tool had a bug. Facebook, WhatsApp and Instagram all go dark.

Now the twist. DNS is the internet's address book. Facebook's DNS servers can't reach the data centers anymore. They're built for that. They stop announcing themselves, through BGP. So they're still running. But nobody on the internet can find them.

But here's the catch. How do you fix it remotely? Normal network access? Down. The backup, out-of-band access? Down too. And without DNS, many internal tools broke.

So Meta sends engineers to the data centers. In person. Slow going. The sites are built to be hard to get into. And the routers are hard to change. Even standing right next to them.

About 21:28 UTC, Facebook is reconnected. Meanwhile, public DNS resolvers had handled 30 times more queries than usual. But don't flip it all on. Power dips hit tens of megawatts. So Meta brings services back gradually. Its storm drills had prepared it for this.

Your recovery path can't depend on the thing that broke. Keep a way in that survives a total network loss. And practise the restart. Not just the failure.

Full breakdown in the lecture.