S3 said 200 OK. Then the read said 404.

transcript

You upload a file. S3 says 200 OK. You read it back. 404 Not Found.

Before 2020, metadata was cached on many front-end nodes. For speed. And it replicated to them asynchronously. Your read hits a node that hasn't heard of it yet. So: 404. The bytes were fine. The map wasn't.

Data lakes missed files. People added sleeps before reading. Or kept their own database of what they'd written. For a storage system!

The fix, in 2020: writes update metadata synchronously. And every read asks a tiny record, the witness. Is my cache current? Usually yes. Use the cache. The check is tiny. If not, read a majority of replicas. Any two majorities overlap. It can't miss the write.

Before: 200, then 404. Now: 200, then version 2. Every time. Same throughput. No price penalty. Strong consistency.

AWS even checked the protocol with formal proofs. Strong consistency, for the price of one tiny check.

Full breakdown in the lecture.