Three months, said the tool. Discord did it in nine days.

transcript

Discord moved trillions of messages to a new database. The first tool said three months. It took nine days.

Early 2022. Discord's messages sit on 177 Cassandra servers. Each channel's messages live together. One channel gets busy. All its reads hit the same few machines. They slow down. And so does every other request routed there.

No panic, though. By 2020, every other database was on ScyllaDB. Then a shield in Rust. Many people, one message? One query. Then new messages go to ScyllaDB. History gets copied behind them.

The first copying tool estimated three months. So they rewrote it in Rust. Their words: meme-driven engineering. Up to 3.2 million messages a second. The whole copy took nine days.

But here's the catch. At the very end, it got stuck. Tombstones. Markers for deleted data Cassandra never cleaned up. They cleared them. May 2022, they switched over. With cake.

Before: 177 servers. Old-message reads, up to 125 milliseconds at p99. After: 72 servers. Reads at 15 milliseconds. Writes? A steady 5 milliseconds.

Migrations that look sudden are usually the last step of a plan. Small databases first. A shield in front. New writes. Then history.

Full breakdown in the lecture.