A 43-second blip moved GitHub's databases across the country

transcript

The network came back in 43 seconds. GitHub stayed degraded for 24 hours. Because in those 43 seconds, its databases moved across the country.

GitHub's data lives in MySQL. One primary takes all the writes. Replicas copy it and serve the reads. A tool called Orchestrator watches. Lose the primary? It promotes a new one. Its nodes vote. A majority must agree.

22:52 UTC. Maintenance cuts a link to the main East Coast data center. Nodes in the West and an East Coast cloud form a majority. They fail over. Writes now go to the West Coast. 43 seconds in, the link is back. Too late.

But here's the catch. The East held seconds of writes the West never got. Now the West takes new writes the East doesn't have. Neither copy is complete. So you can't just switch back. And East Coast apps now talk across the country. The delay breaks them.

So GitHub chose data over speed. It paused webhooks and Pages builds. Then restored the East from backups. Multiple terabytes. That took hours.

Meanwhile, over 5 million webhook events piled up. 80,000 Pages builds waited too. About 200,000 webhook payloads outlived a time limit. Dropped.

Green at 23:03 UTC, the next day. No user data lost. And Orchestrator? GitHub said it behaved as configured. The apps just couldn't live with primaries in another region.

GitHub's first fix: no promoting primaries across regions. Automatic failover is only safe if you can live where it lands. So test the partition. Before it happens for real.

Full breakdown in the lecture.