Counting at scale: Bloom filters, HyperLogLog, Count-Min and top-k
Why exact counting breaks when every distinct key costs memory per slice and per day, and the sketches that count in fixed memory with a known, one-directional error: Bloom filters for membership (sizing, over-filling, deletes, LSM reads, Akamai's cache-on-second-hit, a dedup gate that loses nothing), HyperLogLog for distinct counts (registers, error, HLL++, merging, Redis and BigQuery), Count-Min for frequencies, Misra-Gries and Space-Saving for top-k over minute panes, and where each sits in the week's pipeline.
01 · Previously
W08 · DAY 06 · Counting at Scale: Bloom Filters, HyperLogLog, Count-Min Sketch and Top-K
Fast queries. Now count without remembering.
Previously: the query layer
Previously, on Day 5, you built the query layer: partitions, rollups and caches.
0:00 / 23:08