Cloudflare's control plane, the dashboard and APIs customers use to configure its services, ran mainly in three data centers near Hillsboro, Oregon. They were independent, far enough apart to dodge one disaster, and kept in sync so that if any one went offline, the other two carried on.
On 2 November 2023, the largest of the three, PDX-04, run by Flexential, lost power. One utility feed was cut by maintenance. Then, at about 11:40 UTC, according to Cloudflare, a ground fault on a transformer tripped protections that also shut down the building's generators. The backup batteries were meant to last about 10 minutes; by what Cloudflare saw from its own equipment, they started failing after 4. Cloudflare first knew something was wrong when the facility's two routers went offline at 11:44. Flexential's first message came at 12:28.
Most of the high-availability cluster did fail over to the other two sites. But the surprise was inside: some services that were supposed to survive depended on Kafka and ClickHouse, which ran only in PDX-04. Some newer products had never been moved onto the high-availability cluster at all. Cloudflare had tested losing each of the other two sites, and losing PDX-04's high-availability part. It had never tested losing the whole of PDX-04.
At 13:40 it decided to fail over to disaster recovery sites in Europe; by 17:57 most customers were no longer directly affected. Through all of it, traffic through Cloudflare's network was not affected. Full recovery took until 4 November at 04:25 UTC, after a full bootstrap of the facility, with each server taking between 10 minutes and 2 hours to rebuild.
Cloudflare called a "Code Orange": all non-critical engineering shifted to control-plane reliability, every generally available product would need the high-availability cluster and a tested recovery plan, and chaos tests would remove each core facility entirely.
- Batteries meant to last about 10 minutes started failing after 4
- 11:44 UTC routers offline; Flexential's first message at 12:28
- Most customers no longer directly affected by 17:57, after failover to Europe
- Fully restored 4 November at 04:25 UTC, about 41 hours after it began
Why it matters: Redundancy you haven't tested is a guess. Pull the plug on a whole site on purpose, because that is how you find the one service that only lives there.