A 43-second network blip gave GitHub 24 hours of trouble.

GitHub keeps data like pull requests, issues and logins in MySQL database clusters. Each cluster has one primary that takes all the writes, plus copies, called replicas, that serve reads. A tool called Orchestrator watches the clusters and, if a primary is lost, promotes a new one automatically. Its nodes vote, using a consensus method called Raft, so a majority must agree.

At 22:52 UTC on 21 October 2018, maintenance to replace failing network equipment cut the link between GitHub's US East Coast network hub and its main East Coast data center. It was restored in 43 seconds. In that window, Orchestrator nodes on the West Coast and in an East Coast public cloud formed a majority and failed the clusters over, so writes now went to the West Coast.

That was a trap. The East Coast databases held a few seconds of writes that never reached the West. The West now took new writes the East didn't have. Neither copy was complete, so GitHub couldn't simply switch back. And East Coast apps, now talking to databases across the country, couldn't cope with the extra delay; for many users the site was unusable.

GitHub chose data over speed. It paused webhooks and GitHub Pages builds and restored the East Coast clusters from backups, which took hours for multiple terabytes. Replicas then lagged hours behind as users in Europe and the US started their workday, so people saw out-of-date data until GitHub added more replicas. The site went green at 23:03 UTC on 22 October. No user data was lost, though a few seconds of writes needed manual reconciliation.

GitHub said Orchestrator "behaved as configured", but the apps couldn't support primaries in another region. Its first fix on the list: configure Orchestrator so it can't promote primaries across regional boundaries.

Why it matters: Automatic failover is only safe if the rest of the system can live with where it fails over to. Decide which moves are allowed, and test the partition before it happens for real.