100 petabytes of index, every search under a second

transcript

12,000 searches a second. 100 petabytes of index. Every answer in under a second. Impossible?

1.5 billion mailboxes. About 100 petabytes of index, on these assumptions. About 12,000 searches a second at peak.

But here's the catch. Nobody searches anyone else's mail. So why build one giant index?

One global index? Every query filters 100 petabytes down to you. Instead, every mailbox gets its own index. About 66 megabytes. It lives on your own shard. One shard per query. No fan-out.

Meera searches twelve years of mail. About 60,000 messages. Start with the rarest term. A few hundred messages from Ravi. Then the attachment flag. Then the date. Then the words, form and 16. The set only shrinks.

She labels 40 threads ITR 2026. What gets re-indexed? Nothing. Labels are bitmaps in metadata, applied at query time.

Partition search by user. One huge search problem becomes a billion small ones.

Full breakdown in the lecture.