Your incremental batch job will never count this late event

transcript

This event arrived safely. It's sitting in your data lake. Your batch job will never count it. Not tonight. Not ever.

A full recompute reads every event for the day, every run. Expensive. So you go incremental. Read only events after the last watermark. Then merge them into the existing totals. Much less work.

But here's the trap. It filters on event time. When the event happened. Not when it arrived.

Run one finishes. The watermark moves forward. Then a phone comes back online. Its event is stamped before that watermark. Run two reads only events after the watermark. So it skips this one. And every run after that skips it too.

The full recompute of that date? It reads every event, whenever it arrived. Slower. But the late event is included.

Incremental jobs save work by trusting the watermark. Late data doesn't respect it. It slips past every run.

Full breakdown in the lecture.