~learn.
0:00 / 1:10
W01 · Data at scale

12 partitions. It's December. 11 of them sit idle.

transcript

You split your data across 12 partitions. Nicely scaled. Then December arrives. 11 of them stop taking writes.

The data is time series. One partition per month. Sounds tidy. January to November? Old months. Nobody writes there anymore. Read-only. December takes every single write. One machine doing all the work.

Same trap with user IDs. Ranges of a million each, auto-incrementing. Your app goes viral. Thousands sign up a minute. Every new user gets the highest ID. So every new write hits the last range. It's overloaded. And the other partitions? Sitting idle.

But range isn't wrong. Want users 1 point 5 to 1 point 6 million? One partition. It only breaks when everyone hits the same end. Hash spreads writes evenly instead. But then URLs from the last hour means asking every partition.

Range works when access is spread across the range. It fails badly at one end. The newest ID. The current month. The latest timestamp.

Full breakdown in the lecture.