"Retry 1 of 3", logged forever

transcript

Your logs say: retry 1 of 3. Then retry 1 of 3. Then retry 1 of 3. Forever. The message never reaches the dead letter queue. Why?

A message fails. Consumer A counts: retry 1. And republishes it. Consumer B picks it up. It starts counting from zero. Retry 1. And consumer C? Retry 1. The count lives in each consumer's memory. So the limit never triggers.

The fix: write the retry count into a message header. Read it, increment it, republish it. Now every consumer sees the real count. At the limit, it goes to the DLQ.

And how many retries? It depends on the failure. A missing field fails the same way every time. Zero retries. A poison pill that crashes the consumer? Zero. Quarantine it, page on-call. A timeout or a blip? 5 retries, with exponential backoff. Another service returning 503? 10, with linear backoff.

So why not 15 retries, to be safe? Too few, sure, and blips land in the DLQ. But 15 means every hopeless message burns 15 attempts, blocking everything behind it. More retries isn't safer. It's just slower to quarantine.

Retry what can recover. Quarantine what can't. And keep the count on the message, not in the consumer.

Full breakdown in the lecture.