AWS's hidden internal network jammed, and took its own monitoring down with it.

Customer servers at AWS run on the main AWS network. But behind it sits a second, internal network for the foundations: monitoring, internal DNS, authorization, and parts of the EC2 control plane (the system that launches and manages servers). A set of networking devices links the two.

On 7 December 2021 at 7:30 a.m. PST, in us-east-1, an automated job scaled up one AWS service on the main network. That triggered "an unexpected behavior from a large number of clients" on the internal network: a surge of new connections that overwhelmed the devices between the two networks. Requests got slow and failed, so clients retried, which made more traffic, which made things slower. The congestion locked itself in.

The cruel part: the same congestion cut off the real-time monitoring AWS's operators rely on. They worked from logs instead, spotted internal DNS errors, and moved DNS traffic off the jammed paths by 9:28 a.m. That helped, but didn't end it. AWS's internal deployment systems were on the internal network too, which slowed every fix. And because many customer workloads were still running fine, the team moved carefully so as not to break them.

Congestion eased by 1:34 p.m. and all network devices had recovered by 2:22 p.m. Running EC2 servers had kept working throughout, but much that needed the control plane struggled: launching new servers (back at 2:40 p.m.), Route 53 DNS changes, console logins, API Gateway. AWS's support system relied on the internal network too, so creating support cases was impaired from 7:33 a.m. to 2:25 p.m.

AWS found that its clients' back-off logic, meant to slow retries during congestion, had a latent bug in a code path "in production for many years". It disabled the scaling activity and set about fixing the back-off.

Why it matters: Retries without real back-off turn a slowdown into a jam that won't clear. And keep the tools you debug with off the system you're debugging.