Google's network has a control plane: software that works out routes and tells the network hardware where to send traffic. It runs as jobs under several separate copies of Google's cluster management software in each location, so one copy failing changes nothing. That software also automates data center maintenance. During maintenance, each job is labelled to either move elsewhere or stop and wait.
Google's report says "two normally-benign misconfigurations, and a specific software bug" lined up. The network control jobs were set to stop during maintenance. The cluster software running them was marked as eligible for one particular, rare type of maintenance. And the maintenance software had a bug that let it switch off several clusters at once, even in different physical locations.
On Sunday 2 June 2019 at 11:45 US/Pacific, that rare maintenance began in one location. The automation listed the jobs to stop there, and the list included network control jobs in other locations too. It stopped them all.
The network is designed to "fail static": keep running for a while without the control plane. It did, for several minutes. Then routes between the affected locations were withdrawn and network capacity fell sharply. Users felt it from 11:47. Engineers were alerted two minutes after it began, but their tools were fighting for the same congested network; the backup plans included travelling to secure facilities.
By 13:01 the cause was known and the automation halted. But with every control job stopped in several places, their configuration was lost and had to be rebuilt. It began to roll out at 14:03; capacity returned from 15:19 and full service at 16:10. Google Cloud, G Suite and YouTube were all hit. In us-west2, Cloud Storage requests averaged a 96.2% error rate.
Google's fixes: maintenance that can't touch several locations at once, saved configuration for faster recovery, and a longer fail-static window.
- Elevated packet loss lasting between 3 hours 19 minutes and 4 hours 25 minutes, by region
- Engineers alerted 2 minutes after it began; root cause at 13:01
- Full service back at 16:10 US/Pacific
- 96.2% average Cloud Storage error rate in us-west2
Why it matters: Redundancy only helps if the copies can't fail together. Automation that can touch many locations in one action is a single point of failure, however many copies it touches.