Facebook's data centers are linked by its backbone: its own network of "tens of thousands of miles" of fiber-optic cable. Engineers often take parts of it offline for maintenance.
On 4 October 2021, during one of those routine jobs, a command was issued to check how much backbone capacity was available. Instead, Meta wrote, it "unintentionally took down all the connections in our backbone network". A tool was meant to audit commands like this and stop mistakes, but a bug in it let the command through. Every data center was cut off.
That caused a second problem. DNS is the internet's address book: it turns a name like facebook.com into a server address. Facebook's DNS servers are designed to stop announcing themselves to the internet (through a protocol called BGP) if they can't reach the data centers, because that signals a bad network. With the whole backbone gone, they all did. The DNS servers were still running, but nobody on the internet could find them. Cloudflare saw a burst of routing changes from Facebook at about 15:40 UTC; Facebook, WhatsApp and Instagram all went dark.
Fixing it was hard. Normal and backup ("out-of-band") network access was down, and the loss of DNS broke many internal tools. So Meta sent engineers to the data centers in person. That took time: the sites are built to be hard to get into, and the routers hard to change even when you're standing next to them.
Cloudflare saw Facebook reconnected at about 21:28 UTC. Meta then brought services back gradually: data centers had seen power dips of "tens of megawatts", and flipping everything on at once risked new crashes. Its "storm" drills, which simulate losing a data center or region, had prepared it for that. Meta said there was no malicious activity and no evidence user data was compromised.
- About 15:40 to 21:28 UTC: nearly six hours off the internet
- Every backbone connection down, from one command
- Public DNS resolvers handled 30 times more queries than usual, Cloudflare said
- Power dips of tens of megawatts in individual data centers
Why it matters: Your recovery path must not depend on the thing that broke. Keep a way in that survives a total network loss, and practise the restart, not just the failure.