How long should a dashboard wait for late events? Measure it.

transcript

Your dashboard closes the 2 o'clock hour. Late events keep arriving. So how long should it wait? Most teams guess. Don't.

Query your history. Received time minus event time. In the lesson's example, half arrive within 2 seconds. 99 percent arrive within 5 minutes. And the slowest? 72 hours. A long, thin tail.

Now pick a tolerance. 30 seconds catches about 90 percent. 5 minutes, about 99. 30 minutes, 99.9. An hour, 99.95. Each step costs more delay, or more memory.

But here's the catch. Tolerance is two knobs: the watermark bound, plus allowed lateness. Put it in the bound, and the window fires later. Your first number is delayed. Put it in allowed lateness, and the window fires on time. Late events then update it. That costs state in memory. Not delay.

Then match the use case. Fraud detection: 30 seconds. Speed beats completeness. Daily revenue: an hour. Late events trigger a recompute. Monthly financial close: 7 days. It must be right, so you wait for them.

And set it per source. Web is 99.9 percent in within a minute. Mobile can take hours. Don't guess your lateness. Measure it, then set it.

Full breakdown in the lecture.