Your retries just quadrupled the load on a failing service

transcript

A service starts struggling. Your code does the helpful thing. It retries. And that's what finishes it off.

10 servers call the same service. A request fails on every one of them. One millisecond later, every server retries. No delay. Three retries each. That's 30 requests landing at the same instant. One original, plus three retries. Up to 4 times the load, on a service already struggling.

So retry politely. Exponential backoff: wait longer before each retry. Jitter: add randomness, so the 10 servers don't retry in lockstep. And a retry budget: a cap on total retries, across all clients.

But here's the catch. Before any retry, ask: what if the first request actually succeeded? Retry a payment that already went through, and you charge twice. So retries need idempotency.

Every retry is extra load, aimed at the service that's already hurting. Back off. Add jitter. Cap it. And make it safe to repeat.

Full breakdown in the lecture.