Monzo added six empty servers. They started answering.

transcript

A bank added six empty servers to its database. And they started answering questions. With no data at all.

Monzo kept its data in Cassandra. 21 servers. Every piece of data on three of them. Every read needs two of the three to agree. A quorum.

13:10. The cluster is near its limits. Six new servers join. A setting should keep them quiet until they're filled. Metrics look fine. 13:14. The support team's chat tool stops working. Some customers get logged out. Some data just isn't there.

14:13. They prove what they thought was impossible. Answers are coming from the new, empty servers. For some data, two of its three owners are now empty. So reads come back empty. Transactions can't be found.

The villain: one Cassandra flag. Auto bootstrap, set to false. They thought it only skipped copying data. It also made new servers join active. Serving right away.

But they tested this. Here's the catch. The test added one server. The two old owners still agreed. Six servers broke the majority.

They pull the new servers out. 8 to 10 minutes each. 15:08, Monzo works again for most. 23:00, everyone.

Adding one node and adding six are different operations. Test the exact change. And know what every flag really does.

Full story in the article.