In 2017 Discord kept billions of messages on 12 Cassandra database servers. By early 2022 the cluster had grown to 177 servers holding trillions, and it had become a constant headache.
The worst problem was hot partitions. Discord stores each channel's messages together. When one channel got very busy, all its reads hit the same few machines. Those machines slowed, and so did every other request routed to them. On top of that, the database kept pausing for garbage collection (memory clean-up), and its background maintenance fell behind.
The move was planned and done in steps. By 2020 Discord had already moved every other database to ScyllaDB, a Cassandra-compatible database written in C++. Only the messages cluster was left. Before touching it, the team put a new layer in front: small "data services" written in Rust. If many people asked for the same message at the same moment, the service sent the database one query, not one per person.
The plan was to send new messages to ScyllaDB from a chosen cutover time, then copy the history behind them. The first copying tool estimated three months. The team rewrote it in Rust (in their words, "meme-driven engineering"), and it ran at up to 3.2 million messages per second. The whole copy took nine days. At the very end it got stuck on a few ranges full of "tombstones", markers for deleted data that Cassandra had never cleaned up. Once they cleared those, the copy was done. In May 2022 they switched over, at a team get-together, with cake.
- 177 Cassandra servers replaced by 72 ScyllaDB servers
- Copy time: estimated three months, actually nine days
- Up to 3.2 million messages copied per second
- p99 time to read old messages: 40–125 ms before, 15 ms after
- p99 time to write a message: 5–70 ms before, a steady 5 ms after
Why it matters: Migrations that look sudden are usually the last step of a long plan: smaller databases first, a protective layer in front, new writes to the new system, then the history.