108 GB of sets, or 11 GB of sketches

transcript

Counting unique visitors took 108 gigabytes. A sketch did it in 11.

Picture a store with 10 million visitors a day. Uniques per category and country: 10,000 cells. Each visitor lands in about 3. That's 30 million entries.

Exact sets cost about 40 bytes an entry. That's 1 point 2 gigabytes. Every single day.

Why not just add up the daily counts? A returning visitor counts once per day. Wrong number. So you keep 90 days of sets. 108 gigabytes.

Enter HyperLogLog. It hashes every visitor id. Registers keep the longest run of leading zeros. Out comes an estimate. 0.81 percent error, in 12 kilobytes.

Exact sets for 90 days: about 108 gigabytes. Sketches: about 11. About tenfold smaller.

But here's the twist. Tenfold isn't the big win. Sketches merge. Any date range is just a merge. A 28-day tile reads about 340 kilobytes. No double counts.

Merge sketches. Never add up unique counts.

Full breakdown in the lecture.