Every hour, on the hour, the scheduler gets hit ten times harder

transcript

Every hour, on the hour, your scheduler gets hit ten times harder. And the average will never warn you.

500 million runs a day. About 5,800 a second, on average. But people love on the hour. Say a sixth of each hour's runs land in minute zero. That minute runs at 58,000 a second. Ten times the average.

Thousands of jobs, due in the same second. Start them all at once, and you hammer every target.

Tool one: look-ahead. Claim runs up to 5 seconds early. The queue's full when the second arrives. Tool two: jitter. A job with a 5-minute window starts somewhere inside it. Tool three: queues per team and priority. Workers pull fairly.

Why fair? One team has 50,000 hourly jobs. Another has 5. With one shared queue, those 5 starve.

The price? Not every job starts on the exact second. Design for the spike. Not the average.

Full breakdown in the lecture.