Reddit upgraded Kubernetes on Pi Day, and one renamed label took the site down for 314 minutes.

On 14 March 2023, Pi Day, just after 19:00 UTC, a Reddit engineer started upgrading one of the company's most important Kubernetes clusters, the one running the legacy part of the site, from version 1.23 to 1.24. Everything looked fine for about two minutes. Then the site came to a halt.

The cluster lost all its metrics, so the team was "flying blind", and its internal DNS stopped resolving. The obvious move was to roll back, but Kubernetes has no supported downgrade: going back means restoring the cluster from a backup. Reddit's restore procedure had been written years earlier and never used on a production cluster, and the estimate was hours of guaranteed downtime. So they tried to fix it forward.

After more than two hours without a fix, they made the hard call and restored. The procedure fought back: written for Docker, not the CRI-O runtime Reddit now used, it had to be rewritten live. AWS ran out of capacity for the control-plane instance type. New control-plane nodes wouldn't join, because the backup had been taken on a different node than the one restored, and its security certificates didn't match the hostname. Then traffic came back in steps, from 1% up to 100%, so cold caches weren't crushed.

Only afterwards did the logs reveal the cause. Years earlier, Reddit had set up Calico route reflectors, a few nodes that share network routes with all the others. They picked those nodes by the label node-role.kubernetes.io/master. Kubernetes had renamed "master" to "control-plane", and 1.24 removed the old label, even from running clusters. No reflectors, no routes. That setting was committed nowhere, and everyone who knew about it had moved on.

Why it matters: A setting nobody remembers still runs in production. Keep every piece of config in code, practise the rollback before you need it, and read the upgrade notes for what a new version removes.