Search all of GitHub with grep? 96 seconds a search

transcript

What if you searched all of GitHub with grep? Every search would take over a minute and a half.

115 terabytes of code. 2,048 cores, each scanning 0.6 gigabytes a second. About 96 seconds. Per search.

The whole cluster manages about 0.01 searches a second. One search at a time, and the whole cluster is busy.

So index substrings instead. Each gram points to a sorted list of documents. Intersect a few lists. Then check only the candidate files.

But here's the trick. Git names every file by its content. Shard by repository? Forks and popular repos pile up. Shard by blob id. Identical files become one document. And hashes spread evenly. No hot shards.

115 terabytes of code. Only 28 terabytes of it unique. The whole index: 25 terabytes.

GitHub built this from scratch, in Rust. Per shard, about 100 milliseconds at p99. Git already named every file by its content. Use the name.

Full breakdown in the lecture.