An earthquake hits. The pages that matter are 5 minutes old.

transcript

An earthquake hits a big city. Millions search at once. The pages that matter were published five minutes ago. They're in no index yet.

The target: breaking news findable within 15 minutes. Rebuild the main index more often? It's hundreds of terabytes. Merging it takes hours. Big segments serve fast, but rebuild slow. News would never make 15 minutes.

A detector sees the query spike, and new names in the news. Feeds, sitemaps and trusted news sources jump the crawl queue. New pages land in a small fresh tier, in memory. In seconds. Later, quietly, the fresh tier merges into the main index.

Now the query hits both. The main index and the fresh tier. Freshness gets boosted. So does source trust. Because this is exactly when rumour and spam try to rank.

But here's the catch. One query is now a huge share of traffic. So cache that answer for a short time. Most of the spike never fans out.

A big index for everything. A small, in-memory tier for right now. Freshness is a tier, not a rebuild.

Full breakdown in the lecture.