Google Search, the whole design.

%% title: Search on one page: crawl and index offline, scatter-gather on the query path
flowchart TD
  U@{ icon: "sd:users", label: "Users<br/>~14B searches a day" } --> FE["Query frontend<br/>spell · tokenize · synonyms · intent · 20 ms"]:::edge
  FE --> QC@{ icon: "sd:cache", label: "Result cache<br/>short TTL · ~30 % hits" }
  FE ==> RT["Root / merger<br/>scatter to every shard · gather local top-k"]:::hot
  RT ==> SH@{ icon: "sd:index", label: "Document-sharded index<br/>thousands of shards · 3–10× replicas" }
  RT --> FT@{ icon: "sd:index", label: "Fresh tier · in memory<br/>priority-crawled news, in minutes" }
  RT ==> RK["Ranker<br/>BM25 · PageRank · freshness · location · clicks<br/>snippets for the top 10"]:::core
  FR@{ icon: "sd:queue", label: "URL frontier<br/>priority · 1 req/s per domain" } --> CW@{ icon: "sd:robot", label: "Crawler workers<br/>robots.txt · async fetch" }
  CW -.->|"new links"| FR
  CW --> PS@{ icon: "sd:database", label: "Page store<br/>~2.5 PB compressed" }
  PS --> IB["Indexer<br/>tokenize · build segments · PageRank"]:::service
  IB -->|"finished segments"| SH
  class U client
  class QC cache
  class SH,FT,PS store
  class FR queue
  class CW service

%% title: Scatter-gather: the slowest shard sets the pace
flowchart TD
  RT["Root · fans out to ~1,000 shards<br/>all fast only ~0.004 % of the time"]:::hot --> S1["Shard 1 · docs 0–1M<br/>intersect postings · local top-k"]:::service
  RT --> S2["Shard 2 · docs 1M–2M<br/>local top-k"]:::service
  RT --> SN["Shard N · primary slow"]:::bad
  SN -->|"no answer in a few ms"| HR["Hedged request to a replica<br/>take whichever answers first"]:::queue
  S1 --> MG["Merge local top-k → ~10,000 candidates<br/>deadline: return what has arrived"]:::ok
  S2 --> MG
  HR --> MG

%% title: A query: "best pizza near me" in under 500 ms
sequenceDiagram
  autonumber
  participant U as User
  participant F as Query frontend
  participant R as Root / merger
  participant S as Index shards (all)
  participant K as Ranker
  U->>F: best pizza near me
  F->>F: spell · tokenize · synonyms · local intent (20 ms)
  alt popular query, cached
    F-->>U: cached results
  else cache miss
    F->>R: parsed query
    R->>S: same query to every shard in parallel
    S-->>R: local top-k each (100 ms budget)
    R->>K: ~10,000 merged candidates
    K->>K: BM25 · PageRank · freshness · location · clicks → top 100 (200 ms)
    K->>K: snippets for the top 10 (50 ms)
    K-->>U: results page (~400 ms)
  end

%% title: Tail latency: one shard is slow
sequenceDiagram
  autonumber
  participant R as Root / merger
  participant P as Shard 7 primary
  participant Q as Shard 7 replica
  R->>P: query
  Note over R: no answer after a few ms
  R->>Q: hedged request, same query
  alt replica answers first
    Q-->>R: local top-k
    Note over R: use it · ignore the late primary
  else both miss the deadline
    Note over R: return without shard 7 · slightly incomplete beats slow
  end

%% title: Breaking news: searchable within 15 minutes
sequenceDiagram
  autonumber
  participant D as News detector
  participant C as Crawler
  participant T as Fresh tier
  participant R as Root / merger
  D->>D: query spike · new entities
  D->>C: trusted sources jump the frontier
  C->>T: index new pages in seconds
  R->>T: query the fresh tier
  R->>R: + main index · boost freshness, trust
  R->>R: cache hot query, short TTL
  Note over T: merged into the main index later

%% title: Core entities
classDiagram
  class Shard {
    +shardId : int
    +docRange : slice of doc IDs
    +replicas : 3 to 10
  }
  class PostingsList {
    +term : string
    +docFrequency : int · for IDF
    +postings : sorted by docId
  }
  class Posting {
    +docId : int
    +termFrequency : int
    +positions : int list
    +size : ~12 B
  }
  class Document {
    +docId : int
    +url : string
    +pageRank : float
    +lastCrawled : timestamp
    +contentFingerprint : hash · dup and spam check
  }
  class CrawlURL {
    +url : string
    +priority : importance + freshness
    +lastCrawl : timestamp
    +depth : int · trap guard
  }
  Shard "1" --> "many" PostingsList : every term for its docs
  PostingsList "1" --> "many" Posting : holds
  Posting "many" --> "1" Document : points to
  CrawlURL "1" --> "0..1" Document : fetched as

%% title: A page's state machine, from link to results
stateDiagram-v2
  [*] --> DISCOVERED: link extracted
  DISCOVERED --> QUEUED: new URL, deduplicated
  QUEUED --> FETCHED: crawled, polite rate
  FETCHED --> FAILED: error or robots.txt
  FETCHED --> REJECTED: spam, duplicate or trap
  FETCHED --> FRESH_TIER: breaking news
  FETCHED --> INDEXED: segment built
  FRESH_TIER --> SERVING: searchable in minutes
  INDEXED --> SERVING: segment pushed to shards
  SERVING --> QUEUED: recrawl due
  FAILED --> [*]
  REJECTED --> [*]
  class SERVING ok
  class FAILED,REJECTED bad
  class QUEUED queue