LinkedIn's Kafka doesn't reject a runaway producer. It just answers slowly.

transcript

One client is flooding your Kafka cluster. You could reject it. LinkedIn's trick is sneakier. The broker just answers it slowly.

LinkedIn moves more than 7 trillion messages a day. From thousands of producers, sharing the same clusters. So one runaway producer can hurt the whole cluster. Everyone else's messages too.

So every client gets a quota, tracked at the broker. For example, 10 megabytes a second of producing. And 50 megabytes a second of consuming. Quotas apply per user and client ID, so critical clients can get more.

Over quota, you could reject. Now the client has errors to handle, and retries to pile on. Instead, the broker delays its response. The client just thinks the broker is slow. So it backs off, all by itself. No errors. No data lost. Just delay.

Here's what's clever. A queue removes the natural brake. Producers never wait. A delayed response puts the brake back. The runaway slows down. The cluster stays up.

The best backpressure doesn't feel like an error. Slow the noisy client down, and everyone else keeps moving.

Full breakdown in the lecture.