Five independent regions. One update. Same hour. All down.

transcript

Five regions. Three different clouds. No network between them. And from 06:00 UTC, they all went down together.

Datadog ships changes carefully. Region by region. Node by node. Anything bad rolls back on its own. But the base OS image had a legacy update channel. Still switched on. It installs updates by itself. No stages. No rollback.

8 March 2023, 06:00 UTC. A systemd security update lands. Networking restarts. And deletes the routes containers use to talk. The node drops off the network. A fresh or rebooted node never does this. No obvious test catches it.

So why every region at once? Here's the twist. The OS's default update window. 06:00 to 07:00 UTC. Same clock, every region. That was the hidden link.

US1, EU1, US3, US4, US5. All down. Tens of thousands of nodes offline in one hour. Up to 750 responders. Four shifts.

Live data and alerts first. Old data later. Web access came back at 09:13. Every service, the next morning. History, two days later.

Datadog switched the old channel off. And fixed networkd. Anything that changes your machines rolls out in stages. Even an OS default.

Full story in the article.