Two rules moved below REJECT-THE-REST. Cloudflare lost 19 data centers.

transcript

Cloudflare lost 4 percent of its network. And half of all its requests went with it.

For 18 months, Cloudflare rebuilt its busiest sites on a sturdier design. 19 data centers. London. Tokyo. Mumbai. Frankfurt. Built to survive maintenance. Not this.

A router decides what to tell the internet with a list of rules. The rules are checked in order. Advertise this. Advertise that. And one rule is named exactly what it does. Reject the rest.

21 June 2022. A tidy-up change. Dry run, staged plan, peer reviewed. But it moved two rules from the top to the bottom. Behind reject the rest. In the diff? Just exclamation marks. At a glance, harmless.

06:27 UTC. The rollout reaches the 19 sites. Those addresses stop being advertised. Servers can't reach customers' origins. Engineers can't reach the sites directly. The internal load balancer breaks. Small clusters get flooded.

Through backup access, they find the cause at 06:58. Then the twist. Engineers walk over each other's changes. Reverting the reverts. The last revert lands at 07:42.

The staged rollout reached those 19 sites only at its final step. In a rule list, order is logic. Review a reorder like code.

Full story in the article.