Stripe's database couldn't elect a leader, and the rollback meant to fix it caused a second, longer outage.

Stripe splits its data into database clusters of many shards, each with several redundant copies. One copy, the primary, takes the writes. If it fails, the others hold an election and pick a new one. Stripe exercises failover routinely.

Three months earlier, Stripe had upgraded its databases to a new minor version: tested, rolled out in phases, and fine through many failovers since. But the new version had a subtle fault that only showed up when several nodes were stalled at once.

On 10 July 2019 at 16:35 UTC, one shard hit exactly that state. Its primary failed, and the shard could not elect a new one. Without a primary it couldn't accept writes, and because that shard was used by many applications, including the API, the waiting requests starved the API of compute. Most API requests failed. Restarting every node in the cluster at 17:00 forced an election, and the API recovered at 17:02, 27 minutes in.

Then Stripe did the careful thing. It traced the likely bug to the new version's election code and, at 20:42, rolled back to the previous, known-stable version.

At 21:14 shards went down again. It looked like the same failure, so the team ran the same playbook. It wasn't. The old version clashed with a configuration change made recently to the production shards, and the clash starved them of CPU. The CPU shortage even slowed the fix. The new configuration was rolled out at 22:34, and the API was fully back at 22:47 UTC.

Stripe added alerts for the risky shard state, and said it was adding circuit breakers so one failing cluster can't sink the API, and safer tools for changing config mid-incident.

Why it matters: A rollback is a change too. "Known stable" only means stable with the config it last ran with, so check what has changed around it before you go back.