Your own deploy built a thundering herd, five minutes later

transcript

Your deploy goes out clean. No errors. Graphs look calm. Five minutes later, your database gets stampeded. By that deploy.

At startup, the deploy warms the cache. Every product. Each one with the same TTL. 300 seconds. Five minutes later, every product expires. In the same second. Every request misses. A massive herd, built by your own deploy.

Keys written together, with one TTL, expire together. Double the TTL? That just moves the spike later. It doesn't break the sync.

The fix: randomise every TTL. Plus or minus 30 seconds. Now the expiries spread over a whole minute. Or use plus or minus 10 percent. No maths. No coordination. Just a random number.

For one very hot key, go further. Let requests refresh it early. Each request rolls a small random chance. At 55 seconds, one wins. It refreshes in the background. At 60, nothing expires. Nobody misses. Refreshes cluster just before expiry. And slow recomputes start earlier.

A herd needs everyone to miss at the same moment. So never let your keys expire in lockstep.

Full breakdown in the lecture.