For 18 months, Cloudflare had been rebuilding its busiest locations on a sturdier design it calls Multi-Colo PoP, or MCP, which adds an extra layer of routers ("spines") so parts of a data center can be switched off for maintenance without hurting traffic. By June 2022, 19 data centers, including London, Tokyo, Mumbai and Frankfurt, ran it.
Networks tell the internet which addresses they can reach using BGP. What a router advertises is set by
a policy: a list of rules ("terms") checked in order. One rule in Cloudflare's policy was named exactly
what it did: REJECT-THE-REST.
On 21 June 2022, Cloudflare rolled out a tidy-up change that added labels to some advertisements. It
had a dry run, a staged rollout plan and review by several engineers. But on the spines, the change also
reordered the terms. The two rules that advertise "site-local" addresses moved from the top to the
bottom, behind REJECT-THE-REST. In the diff, a reorder shows only as exclamation marks in front of the terms, so at a glance it looked
harmless.
At 06:27 UTC the rollout reached the MCP locations, and it "swiftly took these 19 locations offline". Those addresses were how Cloudflare's servers reached customers' origin servers, and how its engineers reached those sites directly. Its internal load balancer broke too, so small clusters got as much traffic as big ones and overloaded. The staged rollout hadn't helped: no MCP site was in it until the final step.
Using backup access, engineers found the root cause at 06:58. The last revert finished at 07:42, slowed because engineers "walked over each other's changes, reverting the previous reverts".
Cloudflare's fixes: MCP sites earlier in the stagger, policies that can't be misordered by accident, and an automatic "commit-confirm" rollback.
- 19 data centers, about 4% of Cloudflare's network
- 50% of total requests affected
- 06:27 UTC start; root cause found at 06:58; last revert at 07:42
- 18 months spent moving those sites to the new design
Why it matters: In rule lists, order is logic. Review a reorder like a code change, put your riskiest sites early in a rollout, and have changes roll themselves back unless someone confirms they worked.