AWS's internal network jammed, and took its own monitoring with it

transcript

AWS had an outage. Then it lost the screens to see it. Because the jam hit the network the monitoring lived on.

AWS runs your servers on its main network. But behind it hides an internal network. Monitoring, DNS, parts of the control plane.

7 December 2021. 7:30 a.m. An automated job scales up one service. Clients on the internal network flood the link devices with connections. Requests slow down and fail. So clients retry. More retries, more load, more failures. The jam locks itself in.

But here's the catch. What else lived on that network? The live monitoring. Operators had to read logs instead. The deploy tools, to push fixes. Even support cases.

From logs, they spot DNS errors. By 9:28, DNS is moved off the jam. It helps. It doesn't end it. The network recovers at 2:22 p.m. New server launches only work again at 2:40.

The villain? The clients had back-off logic. Slow down when it's busy. A latent bug kept it from backing off enough. In code that ran for many years.

Retries without back-off turn a slowdown into a jam. And never keep your monitoring on the system it monitors.

Full story in the article.