One renamed label took Reddit down for 314 minutes.

transcript

Pi Day, 2023. Reddit goes dark for 314 minutes. The villain? One word that quietly disappeared.

Just after 19:00. An engineer upgrades Reddit's legacy Kubernetes cluster. Two minutes later, the whole site screeches to a halt. Metrics gone. Internal DNS broken. They're flying blind.

Plan A is always roll back. But here's the catch. Kubernetes has no supported downgrade. You restore from backup. Their restore had never run on a production cluster. Estimate: hours down.

Two hours in, no fix. They pull the ripcord and restore. The guide was written for Docker. They rewrite it live. AWS runs out of their server type. New nodes won't join. The backup came from a different node. Its certificates don't match.

Finally it's healthy. But they don't flip traffic on. One percent. Then twenty. Then fifty-five. Then everyone.

The cause came later, in the logs. Calico route reflectors. They found their nodes by one label. master. Kubernetes 1.24 removed that label. Even from running clusters. No reflectors. No routes. Nothing could talk.

That setting was committed nowhere. Everyone who knew had moved on. Put every setting in code. And practise the rollback first.

Full story in the article.