Three servers, one 10 AM job, and three locks that each break

transcript

One cron job. Run the report at 10. You have three servers. At 10 AM, all three say: my job! It runs three times. So, add a lock?

Fix one: a lock in the database. Server 1 grabs it and starts. Five seconds later, Server 1 loses power. Still holding the lock. Server 2 tries. Blocked. Nothing ever clears it. The job never runs again, until a human steps in.

Fix two: the lock expires after 5 minutes. A dead server can't block forever. But today's report takes 8. At 10:05 the lock expires mid-job. Server 2 grabs it and starts the same job. At 10:08 Server 1 finishes, and releases the lock. Server 2's lock. The job ran twice. The lock state is garbage. Expiry fixed dead servers, and broke slow ones.

Fix three, in Redis: only delete the lock if it's still yours. Check with GET, then DELETE. 10 milliseconds before expiry, Server 1 checks. Still mine. The lock expires. 5 milliseconds later, Server 2 takes it. Then Server 1's DELETE lands. It deletes Server 2's lock.

See the pattern? Every fix opens a new hole. So stop locking per job. Elect one leader on a lease. Give every leader a bigger token, and reject the old ones.

A lock with no expiry deadlocks. A lock with expiry double-runs. So make the job idempotent too. It will run twice someday.

Full breakdown in the lecture.