Cloudflare could survive losing a data center. Except the one it never tested.

transcript

Cloudflare built its control plane to survive losing a data center. Then it lost one. And the dashboard went down anyway.

The dashboard and APIs run in three data centers in Oregon. All kept in sync. Lose any one, the other two carry on. The biggest is PDX-04. Remember that name.

2 November 2023. Maintenance cuts one power feed to PDX-04. About 11:40. A ground fault. The protection shuts the generators off too. Batteries meant to last 10 minutes start failing after 4. 11:44. The routers go dark. The provider's first message comes at 12:28.

No problem, right? Two sites left. But here's the twist. Services meant to survive depended on Kafka and ClickHouse. And those ran in only one place. PDX-04.

Cloudflare had tested losing each of the other two sites. And losing part of PDX-04. Never the whole building.

13:40. They decide to fail over to disaster recovery sites in Europe. 17:57. Most customers are no longer directly affected. Full recovery? About 41 hours after it began.

Cloudflare called a Code Orange. Its plan: chaos tests that remove whole facilities. Pull the plug on a whole site, on purpose. Before a bad day does it for you.

Full story in the article.