Your idempotency key expired before the payment finished
transcript
You did everything right. Every payment carries an idempotency key. The customer still got charged twice. Because the server forgot the key.
Watch one slow payment. Sent at zero. The key is recorded. At 3 point 5 seconds, the bank call times out. The client retries at 5. Same key. Caught. Good. And at 31 seconds, the original finally completes at the bank. Very slow. But real.
Keep the record for 60 seconds, and the completion at 31 is recorded. Keep it for 10 seconds? The 5 second retry is still caught. But at 31, the record is gone. An orphan charge, with no record. And the next retry? Nothing stops it. It can charge again.
The rule: keep records at least 10 times your longest retry window. A 30 second timeout, retried 3 times, is a 90 second window. So at least 15 minutes. For payments, 24 hours. Storage is cheap. Duplicate charges aren't.
But here's the catch. An extreme request finishes at 23 hours 59 minutes. The TTL deletes the record a minute later. The client retries. Charged twice, a day apart. So extend the TTL when a request completes.
An idempotency key only protects you while you remember it. So remember longer than anyone will retry. Then a little longer.
Full breakdown in the lecture.