Under overload, the right move is to throw data away. Some of it.

transcript

Your pipeline is drowning. Millions of events behind. The smartest move? Throw some of them away. On purpose.

That's load shedding. Every message carries a priority. Low, like click events, goes first, from the warning level. Medium goes at critical. High, only in an emergency. And critical, payments and orders, is never shed. Not even in an emergency.

Now an illustrative surge-pricing pipeline. New Year's Eve, 10 times normal traffic. An event older than 30 seconds? Dropped. Pricing needs now, not history. Lag over 10 seconds? Keep 1 in 10 location updates. But every ride request. And each city runs its own pipeline. New York's overload never touches Los Angeles.

Why throw data away? Stale prices mean wrong driver incentives. A rough price now beats an exact one from 5 minutes ago.

But here's the catch. This only works when old data is worthless. Netflix encoding can't skip a job. A skipped encode is a missing video. So Netflix never sheds. It uses priority queues and scaling, and accepts delays at peak.

Better to lose some metrics than crash, and take the payments down with them. Shed what you can afford to lose. Protect what you can't.

Full breakdown in the lecture.