100 million events a day, and not one of them is an answer

transcript

Your platform logs 100 million events a day. And not one of them answers a single question.

Every visitor leaves a trail. 50 million page views a day. 20 million clicks. 5 million add to carts. And half a million purchases. The business doesn't want rows. It wants answers.

What's our conversion rate, this hour? Which products are trending, right now? And the flash sale revenue. Live. While over 1,000 events a second pour in.

Now the twist. A user taps something in the subway. No signal. Five minutes later, your dashboard says 1,000 events. At 30 minutes, they come up, and the event is finally sent. Now the dashboard says 1,050. Someone asks: why did the number change?

And one dashboard asks for daily revenue, over a year. A billion rows. 60 seconds to draw one chart. Unacceptable.

Five days. Capture everything. Stream or batch. Model it. Late data. Sub-second dashboards. Data is useless until you refine it. This week, you build the refinery.

Full breakdown in the lecture.