A million messages a second, and speed isn't the hard part

transcript

A million messages a second. Sounds like a speed problem. It isn't. Speed is the easy part.

A million messages at 500 bytes. 500 megabytes a second in. Three replicas each. 1 point 5 gigabytes a second to disk. Three groups read it all. Another full read each. Not another copy.

Keep it for 7 days. About 302 terabytes. Times 3 replicas. About 900 terabytes of disk. About 20 terabytes a broker. That's 45. Call it 50, with headroom.

Now spread the writes over 50 brokers. About 30 megabytes a second each. That's nothing.

Every write is an append. Sequential, the fastest thing a disk does. Caught-up readers get recent data straight from the page cache. And sendfile ships it to the socket. No copy through the broker.

So storage sets the broker count. Not throughput. A log is the friendliest thing you can do to a disk.

Full breakdown in the lecture.