The worker froze mid-job. Then it woke up.

transcript

2 a.m. A worker starts billing your customers. Then it freezes. Mid-job.

Can a scheduler promise exactly once? No. The work happens in someone else's system. Over a network. A worker calls, then dies. Did the call work? Nobody knows. So promise at least once. With one active attempt at a time.

Attempt 1 calls billing. Then a long garbage collection pause. No heartbeats. After 60 seconds, its lease expires. Attempt 2 starts. Attempt 2 calls billing with the same run key. Billing has seen it, and returns the first result. Then the zombie wakes up and tries to write its result. Rejected.

Leases. Heartbeat every 20 seconds, expire after 60. Slow but alive keeps its lease. A run key: the job ID plus its scheduled time. Retries share it. And attempt numbers. Once attempt 2 exists, attempt 1 can't write.

You can't make a double run impossible. So make it rare. And make it harmless.

Full breakdown in the lecture.