You added a cache node. The database went down.

transcript

Your cache is busy. So you add one more node. Then your database falls over. Because you added capacity.

20 cache nodes. Each key goes to hash mod 20. You add node 21. Now every key goes to hash mod 21. A key stays put only if both match. About 1 in 21. So about 95 percent move.

Those keys now point at nodes that don't have them. So 95 percent of reads miss. At once. Nearly the whole read load lands on the database. Adding capacity just caused an outage.

The fix: stop tying keys to the node count. Hash keys onto a fixed space. Map that space to nodes. Now a new node takes about 1 key in 21. About 5 percent.

One way: 16,384 fixed slots. The scheme Redis Cluster uses. Every key hashes to a slot. Each node owns a range of slots. Add a node? Move a few slots to it, key by key. You can even pause it.

Hash mod N ties every key to the number of nodes. Change N, lose the cache. Hash to slots. Then a new node moves 5 percent, not 95.

Full breakdown in the lecture.