A maintenance job switched off Google's network brains in several places at once

transcript

Google ran many copies of its network brains. For safety. One maintenance job switched them off. In several places at once.

Google's report blames three things lining up. Network control jobs were set to stop during maintenance. They were eligible for one rare kind of maintenance. And a bug let it stop clusters in different locations. At once.

2 June 2019. 11:45. That rare maintenance starts in one location. The list of jobs to stop includes network control in other locations. The network keeps running without its brains. For several minutes. Then routes vanish. Capacity falls sharply.

Engineers were alerted in two minutes. But here's the catch. Their tools were fighting for the same jammed network. Backup plans included travelling to secure facilities.

13:01. Cause found. Automation halted. But every control job in those places had stopped. Their configuration was lost. Rebuild it. Capacity returns at 15:19. Full service at 16:10.

Packet loss for three to four and a half hours, depending on region. In one region, Cloud Storage errors averaged 96 point 2 percent. Google Cloud. G Suite. YouTube. All hit.

Redundancy only works if the copies can't fail together. One action that touches every copy is a single point of failure.

Full story in the article.