GitHub's runners got stuck retrying revoked jobs: the backlog that fed itself

transcript

GitHub Actions runs 124 million jobs a day, on average. In August 2026, its runners got stuck. On jobs already revoked.

That's 1,440 jobs a second. Each on a fresh virtual machine, destroyed afterwards. At an assumed five minutes a job, about 430,000 VMs busy at once.

6 August. A routine deployment briefly cuts capacity. During recovery, runners are handed jobs already revoked. They get stuck retrying them. The backlog feeds itself.

Here's the question. How does a runner know its job is still its job? The quest's answer: a lease.

Give every assignment a lease with an epoch. The runner shows that epoch to start and report. Current? Run it. Lease lapsed? New epoch. The old runner is refused at once.

Without it, a stale runner keeps retrying, and the backlog grows. With it, one fast rejection. Then the runner takes new work.

Retries need a way to say no. An epoch is that no.

Full breakdown in the lecture.