One phone offline for two days froze every window on the dashboard

transcript

One phone goes offline for two days. And not a single window on your dashboard closes.

Quick recap. A watermark says: every event up to time W has probably arrived. A window closes when the watermark passes its end.

Users differ. Some are always online. Some are often offline. So you track a watermark per user, per key. The global watermark is the minimum of them all. Safe, right?

Here's the catch. The phone goes silent. Its watermark stops. Everyone else keeps sending. Their watermarks move on. But the minimum is still that phone. Pinned in the past. So the global watermark never reaches a window's end. Nothing closes.

The fix: an idle timeout. A key not seen within the timeout is left out of the minimum. The watermark moves again. One player offline for 2 days can't stall everyone.

A minimum moves only as fast as its slowest key. So leave the sleeping ones out.

Full breakdown in the lecture.