Monzo kept its data in one Cassandra database cluster of 21 servers. Every piece of data lived on three of them, and every read or write needed two of the three to agree, a "quorum".
Running close to its limits, the cluster needed growing. On 29 July 2019 at 13:10 BST, engineers added six new servers. They believed a setting would keep the new servers inactive until the team copied data onto them, one by one. Metrics looked fine.
Within minutes, things broke in odd places: at 13:14 the support chat stopped, internal tools returned "404", and some customers were logged out. A settings service backed by Cassandra said one key didn't exist, while others were still there.
At 14:13 they proved what they had thought impossible: answers were coming from one of the new, empty servers. The new servers had taken ownership of parts of the data and started serving it without holding any of it. For some data, two or three of its owners were now new servers, so the quorum no longer protected it. Transactions couldn't be found; balances looked wrong.
The culprit was a Cassandra flag, auto_bootstrap. Monzo had set it to false after its last scale-up, in October 2018, so replaced servers would be restored from backups. It turned out the flag also decides whether new servers join active. Monzo's test had added just one server, where the two old owners still agreed.
From 14:18 they removed the new servers one at a time, about 8 to 10 minutes each. By 15:08 Monzo worked again for most customers. Replaying events and reconciling data took until 23:00.
- 21 servers, each piece of data on 3, quorum of 2
- 6 new servers added at 13:10; tested with just 1
- Almost an hour from the first alert to seeing that Cassandra was at fault
- Main impact over at 15:08; all customers fine by 23:00
Why it matters: Test the change you'll actually make: adding one node and adding six aren't the same. Know what each database setting really does before relying on it.