S3 answers 503 Slow Down. Retrying harder makes it worse.

transcript

Your batch job writes 20,000 files a second to S3. S3 answers: 503. Slow Down. What do you do?

Each S3 prefix handles at least 3,500 writes a second. And 5,500 reads. Your job sends 20,000 writes. All under one prefix: logs.

Here's the twist. Behind that 503, S3 has seen the rate. It's splitting that partition by prefix range. It just needs a moment.

Retry in a tight loop? You pour more load onto the very partition that's overloaded. So you get more 503s. Back off instead. Wait 100 milliseconds, then 200, then 400. Up to 5 retries. The partitions split, and the requests succeed.

The lasting fix: spread the work. Logs part 00 to part 15, written in parallel. Each prefix gets its own limits. No single hot spot.

And the old trick of random leading characters? That's advice from before 2018. S3 splits busy prefixes itself now. Not needed.

A 503 here isn't a wall to push through. It's S3 asking for a moment. Back off, then spread out. Slow Down means slow down.

Full breakdown in the lecture.