The endpoint was back at 4 PM. Webhooks kept arriving until 6.

transcript

A partner's server goes down at 2 PM. Back at 4. Healthy. Your webhooks keep trickling in until 6. Two hours late. Why?

2 PM. Event 1 fails. Retry in a minute. It fails at 2:01, 2:06, 2:21, 3:21. Every gap gets longer. That's backoff. At 4, the endpoint is back. But event 1's next retry is already booked. For 5:21.

Meanwhile, 10,000 events piled up for this partner. Each at its own stage of backoff. One from 2:30? Its next try is 5:51. Stragglers keep landing until about 6. The endpoint was healthy the whole time.

Here's the twist. Backoff was right while the endpoint was down. It's wrong the moment it's back. The schedule doesn't know.

The fix. 10 failures in a row? Mark the endpoint unhealthy. A health checker probes it every 30 seconds. The moment it answers, take every pending retry off the schedule. Deliver them now. Still capped by that endpoint's rate limit.

Plain backoff: about 2 hours of stragglers. Probe and flush: all 10,000 delivered by 4:13. And ramp up gently. Don't flood it the second it wakes.

Backoff protects a sick endpoint. Then it punishes a healthy one. So watch for recovery. Then deliver everything you owe.

Full breakdown in the lecture.