422 GB of RAM for 15 queries a second?

transcript

422 gigabytes of RAM. Per replica. To serve 15 queries a second.

Keep every MegaMart conversation for four years. That's about 100 million vectors. Each one is 4,096 bytes of floats, plus 128 of links. That's 422 gigabytes. A few cores could handle the load.

So don't buy the RAM. Shrink the vectors. Product quantisation. Split each vector into 64 pieces.

Each piece is 16 numbers. Swap each piece for the nearest of 256 centroids. That's one byte a piece. 64 bytes. 64 times smaller.

Full precision: 422 gigabytes. 8-bit: 117. 1-bit BBQ: 27. IVF-PQ: 7.2 gigabytes.

But here's the catch. Compressed distances are approximate. So take the top 200. Fetch their full vectors from SSD. Rescore them exactly. At 15 queries a second, that's 12 megabytes a second. Almost nothing.

And today? MegaMart's whole index is 13.5 gigabytes. One node, plus a replica. Compressing it now would only cost recall. So quantise when you outgrow a node. Not before.

Full breakdown in the lecture.