500 events in one upload. 3 are broken. Reject all 500?

transcript

A phone uploads 500 events at once. 3 of them are broken. Do you reject the whole batch?

Phones go offline. So the app buffers events on the device. Back online, it sends them in one batch. Up to 1,000. This time, 500. And 3 fail validation.

Failing the whole batch feels clean. All or nothing. But then one bad event costs 497 good ones.

So each event is validated on its own. The 497 valid ones go to Kafka, in one batch. The 3 invalid ones go to the dead letter topic. The reply reports both. Accepted, rejected, and details for up to 10 errors.

Same idea for time. A phone was offline for weeks. Its events are 40 days old. Over 30 days old gets a warning. But it's accepted. That's real usage. More than 5 minutes in the future? A broken clock. That's an error.

Quarantine the bad events. Keep the good ones. One broken event should never cost you 497 good ones.

Full breakdown in the lecture.