Stripe fixed its outage. Then the fix caused a second one.

transcript

Stripe's API went down. They fixed it. Then the fix took it down again. For longer.

10 July 2019, 16:35 UTC. A database shard's primary fails. The other copies hold an election. And can't pick a winner. A bug in the new version. It only bites when several nodes stall. No primary, no writes. The waiting starves the API.

Most API requests fail. They restart every node in the cluster. That forces an election. 17:02. Back. 27 minutes. Phew.

Now the careful part. They find the likely bug in the election code. So they roll back to the old, known-stable version.

21:14. Shards go down again. Same symptoms. So they run the same playbook. But it's not the same bug. The old version clashed with a recent config change. The shards were starved of CPU. Even the fix crawled.

The bug: 27 minutes down. The fix: 21:14 to 22:47. Ninety-three minutes.

Known stable means stable with the config it last ran with. Before you roll back, check what changed around it.

Full story in the article.