A security update ran at the same hour in every Datadog region, and took them all down.

Datadog built its regions to fail alone: separate zones, three different cloud providers, no direct network links between them and no shared control plane. Its own changes roll out region by region, node by node, with automatic rollback.

On 8 March 2023, at 06:00 UTC, a security update to systemd, the program that starts and manages services on Linux, was applied automatically to many of its virtual machines. It skipped Datadog's careful tooling: the base operating system image still had a legacy security update channel switched on, and it installed the update by itself.

On Ubuntu 22.04, the update restarted systemd-networkd, the part that manages networking. On restart it deleted the network routes that Cilium, the plugin connecting containers, had set up. The nodes went offline. A fresh or rebooted node never showed this, because at boot systemd-networkd starts before those routes exist. No obvious test reproduces it.

Then the twist. The operating system's default window for automatic updates was 06:00 to 07:00 UTC. So it hit the US1, EU1, US3, US4 and US5 regions at the same time. Tens of thousands of nodes dropped in that hour. It was Datadog's first global incident.

Recovery was slow. Some clouds replaced every broken node at once, a "thundering herd" that tested the providers' own rate limits. Datadog put live data and alerts first, then old data. Web access returned at 09:13 UTC, all services were working by 08:58 UTC on 9 March, and backfilling history finished at 06:25 UTC on 10 March.

Datadog then disabled the legacy channel and told systemd-networkd to leave routes alone on restart.

Why it matters: Independent regions can still share a hidden clock. Anything that changes your machines, even an OS default, must go through the same staged rollout as your code.