"Run this report every night at 2 a.m." "Send this reminder in 30 minutes." "Retry this webhook until it works." Cron does it on one machine; this question asks you to do it for a whole company, on a fleet where machines die mid-job. The interviewer is listening for three things: how you find due jobs without scanning everything, how you make sure a job runs when it should, and what you do about the job that runs twice.
The job-scheduler interview end to end: 100 million one-off, delayed and cron jobs, about 5,800 runs a second with 10x spikes at the top of the hour, each started within 5 seconds. Schedulers hold etcd leases on 1,024 partitions and claim due jobs by next_run_at, an outbox feeds Kafka, and workers run each job under a heartbeated lease. Deep dives cover failover with fencing, a timing wheel and look-ahead for the :00 spike, and why runs are at-least-once, made harmless with attempt numbers, a (job_id, scheduled_at) run key and misfire policies.
Phase 1: Requirements (5 minutes)#
"What does a job actually do: run code we host, or call something?"
"Assume it calls an HTTP endpoint or starts a container task owned by another team. We schedule and track it; they own the work."
"And if a job is triggered twice by mistake, how bad is that?"
"For some jobs, very bad: imagine billing customers twice. Tell me what you can guarantee."
- Create jobs that run once at a time, after a delay, or on a cron schedule (
0 2 * * *) - A job targets an HTTP endpoint or a container task, with a timeout
- Retries with exponential backoff, up to a limit; then a dead-letter state
- Pause, resume, delete; see run history and status
- Per-team quotas and priorities
- Out of scope: multi-step workflows (DAGs), which build on top of this
- Timeliness: a run starts within 5 s of its scheduled time
- Reliability: no scheduled run is silently skipped, even if a scheduler dies
- No concurrent double runs: at most one attempt of a run is active at once
- Scale: 100M stored jobs; spikes at :00 when thousands of crons share a minute
- Isolation: one team's flood of jobs can't delay everyone else's
Phase 2: Estimation (5 minutes)#
- Stored job definitions: 100M × ~1 KB (schedule, target, retry policy, owner) = ~100 GB
- Runs: 500M/day ÷ 86,400 s ≈ 5,800/s on average
- Top-of-the-hour spike: people love
0 * * * *. If 10% of an hour's runs share minute :00, that minute runs at about 10x the average, ~58,000/s for a few seconds - Run history: 500M × ~200 B = 100 GB/day → 3 TB for 30 days, then archived
- Each run is a small record plus a call to someone else's system; our side is cheap, theirs may not be
"The average is easy. The design is shaped by the spike at :00 and by the guarantee: thousands of jobs due in the same second, each needing exactly one owner."
Phase 3: API (3 minutes)#
POST /v1/jobs
{ "name": "nightly-invoices", "schedule": { "cron": "0 2 * * *", "timezone": "Asia/Kolkata" },
"target": { "type": "http", "url": "https://billing.internal/run-invoices" },
"timeout_s": 900, "retry": { "max_attempts": 5, "backoff": "exponential" },
"misfire_policy": "run_once" }
→ 201 { "job_id": "j_5521", "next_run_at": "2026-10-26T02:00:00+05:30" }
GET /v1/jobs/j_5521/runs?limit=20 → run history: scheduled_at, attempt, status, duration
POST /v1/jobs/j_5521/pause → 200
DELETE /v1/jobs/j_5521 → 204Every run is identified by (job_id, scheduled_at). That pair is sent to the target as an idempotency key, so the target can tell a retry of the 2 a.m. run from the next night's run.
Phase 4: High-level design (10 minutes)#
%% title: Job scheduler: a store of due times, partitioned schedulers, a queue, and workers with leases
flowchart TD
C@{ icon: "sd:users", label: "Teams · API / UI" }
API["Job API<br/>create · pause · history"]:::service
DB@{ icon: "tech:postgresql", label: "PostgreSQL · jobs<br/>index on next_run_at" }
E@{ icon: "tech:etcd", label: "etcd<br/>partition leases" }
SCH["Schedulers<br/>one owner per partition"]:::core
Q@{ icon: "tech:kafka", label: "Kafka · runnable tasks<br/>by priority / team" }
W@{ icon: "sd:worker", label: "Workers<br/>lease · heartbeat · timeout" }
T@{ icon: "sd:api", label: "Targets<br/>HTTP / container tasks" }
DLQ@{ icon: "sd:queue", label: "Dead-letter runs" }
C --> API
API --> DB
SCH -->|"claim due jobs"| DB
SCH -.->|"hold lease"| E
SCH --> Q
Q --> W
W -->|"call with run key"| T
W -->|"status · retry · next run"| DB
W --> DLQ
class C client
class DB,E store
class Q queue
class W service
class T,DLQ external- Due soon. Jobs are split into 1,024 partitions by
hash(job_id). Each scheduler holds leases for some partitions in etcd and polls its partitions for jobs withnext_run_atin the next few seconds. - Claim and advance. In one transaction it marks the run
QUEUED, writes the run record, and movesnext_run_atto the following night. A crash before commit means nothing happened; after commit, the run is recorded. - Enqueue. It publishes the run to Kafka (via an outbox table, so the queue message can't be lost or duplicated by a crash between the commit and the publish).
- Execute. A worker takes the run, records
RUNNINGwith a lease (say 60 s, renewed by heartbeat), and calls the target with the run key and a timeout. - Finish. Success marks the run
SUCCEEDED. Failure schedules a retry with backoff; after the last attempt the run goes to the dead-letter state and its owner is alerted.
The job definitions and the run state live in the database because that's what must survive anything. The queue is a delivery mechanism, not the source of truth: if it's lost, a reconciler finds runs stuck in QUEUED and re-enqueues them.
Phase 5: Deep dives (15 minutes)#
Deep dive 1: Finding due jobs without scanning 100 million rows#
How does a scheduler find what's due?
A single scheduler loop runs SELECT * FROM jobs WHERE next_run_at <= now() every second. Name two problems at 100M jobs and 58,000 runs a second, and how you'd fix them.
One strong answer
- One loop is a bottleneck and a single point of failure. Split the jobs into partitions and give each partition to one scheduler at a time through a lease (leader election per partition). More schedulers means more partitions in flight; a dead scheduler's leases expire and others take them.
- Two schedulers grabbing the same job. During a lease handover, or if partitions overlap, both could enqueue the same run. Claim with a conditional update (
UPDATE … SET status='QUEUED' WHERE id=$1 AND next_run_at=$2) orSELECT … FOR UPDATE SKIP LOCKED, and carry the lease's fencing token: the database rejects writes from an older lease holder. - Scanning: an index on
(partition, next_run_at)makes "due in the next 5 seconds for my partitions" a short range scan, not a table scan.
For the near future, schedulers can also keep an in-memory timing wheel: a ring of buckets, one per second for the next minute. Every few seconds the scheduler loads jobs due in the next minute from the database into their buckets, and the wheel's hand fires each bucket on time. The database stays the source of truth; the wheel only removes polling latency. It's the same idea Kafka and Netty use for timeouts.
The :00 spike. Thousands of jobs due in the same second can't all start in that second without hammering their targets. Three tools:
- Look-ahead: claim and enqueue runs up to 5 seconds early, so the queue is full when the second arrives.
- Jitter where allowed: jobs that accept it (
"window": "5m") get a random offset inside their window. - Per-team queues and quotas: runs go to a topic by priority and team, and workers pull fairly, so one team's 50,000 hourly jobs can't starve someone else's 5.
Deep dive 2: The job that runs twice#
Exactly-once execution isn't possible when the work happens in someone else's system over a network: if a worker calls the target and then dies, nobody knows whether the call took effect. The honest guarantee is at-least-once, with at most one active attempt at a time, and the tools to make a repeat harmless.
Leases and heartbeats
A worker owns a run only while its lease is fresh. It heartbeats every 20 s; if it dies, the lease expires after 60 s and the run is retried. A slow but alive worker keeps its lease, so a long job isn't duplicated just for being long.
Fencing tokens
Every attempt gets an increasing attempt number. Status updates from attempt 2 are ignored once attempt 3 exists, so a "zombie" worker that wakes up after a GC pause can't overwrite the newer attempt's result.
An idempotency key for the target
(job_id, scheduled_at) goes with every call. The billing service stores the keys it has processed: a second call for the 2 a.m. run returns the first result instead of charging again.
Timeouts on both sides
The worker times out the call; the target should also stop work past the deadline. Otherwise a retry starts while the first attempt is still running.
Missed runs. If the whole scheduler was down from 1:55 to 2:10, did the 2 a.m. run happen? Each job declares a misfire policy: run_once (run it now, once), run_all (catch up every missed occurrence, useful for hourly data loads), or skip (pointless to send yesterday's reminder). Without a policy, a recovering system either silently skips runs or fires a burst of stale ones.
Phase 6: Trade-offs and failure modes (5 minutes)#
A scheduler dies
Its partition leases expire in seconds; other schedulers acquire them and continue from the database. Runs it had claimed are already recorded and queued.
A worker dies mid-run
The lease expires, the run is retried with a higher attempt number, and the target dedupes on the run key.
Kafka loses messages or is down
Run records in QUEUED with no worker after a few minutes are re-enqueued by a reconciler. The database, not the queue, says what must run.
A target is down
Retries back off exponentially with jitter, so thousands of retries don't hammer it in sync. After the last attempt: dead letter and an alert to the owning team.
Clock trouble
Schedulers compare against the database's clock, not their own. Time zones and daylight-saving changes are computed from the job's declared time zone, so 0 2 * * * doesn't run twice when clocks go back.
| Decision | We chose | We gave up |
|---|---|---|
| Finding due jobs | Partitioned schedulers with leases + index on next_run_at | One simple global loop |
| Delivery | Outbox → Kafka, DB as the truth | Fewer moving parts |
| Guarantee | At-least-once + leases + idempotency keys | A (false) promise of exactly-once |
| Spikes | Look-ahead, jitter windows, per-team quotas | Exact-second starts for every job |
Go deeper in the course#
- Week 2, Day 5: Distributed cron and job scheduling: the core lesson for this problem: leader election, execution guarantees, heartbeats and "the leader dies mid-job"
- Week 5, Day 5: Leader election and coordination: leases, etcd and fencing tokens in depth
- Week 2, Day 2: Idempotency in practice: the run key that makes a double run harmless
- Week 2, Day 1: Timeout hell: timeouts on both sides of the call
- Week 3, Day 2: Transactional outbox: enqueueing a run without the dual-write problem
- Week 3, Day 4: Dead letters and poison pills: what happens after the last retry
- Week 9, Day 2: Noisy neighbour prevention: per-team quotas and fair scheduling
- Week 2, Day 4: Webhook delivery system: retries with backoff to endpoints you don't control
Check yourself#
Job scheduler
Q1/4 A worker calls the billing endpoint, then crashes before recording success. The lease expires and the run is retried. What stops a double charge?
Answer & explanation
Answer: C
Nobody can know whether the first call took effect. At-least-once execution plus an idempotency key at the target makes the repeat harmless.
Q2/4 A worker freezes in a long GC pause, its lease expires, attempt 2 starts, and then the first worker wakes up. What keeps its stale result out?
Answer & explanation
Answer: A
Leases alone can't stop a zombie; an increasing attempt number (a fencing token) checked on every write can.
Q3/4 Thousands of jobs are scheduled for 14:00:00 exactly. Which technique does NOT help?
Answer & explanation
Answer: D
Full scans add load at the worst moment. Look-ahead, jitter and fair queues spread or absorb the spike.
Q4/4 The scheduler fleet was down from 13:55 to 14:10. What decides whether the 14:00 run of a job happens?
Answer & explanation
Answer: C
Different jobs need different catch-up behaviour; making it explicit avoids both silent skips and bursts of stale runs.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.