Gmail.

How Do You Say "Yes, I Have It" to the Whole Internet, and Never Be Wrong?

It is the evening of 31 July in Pune, the usual last day to file income tax returns. Meera Iyer, a chartered accountant, has eleven returns left. At 9:40 pm a client in Nashik, Ravi Kulkarni, finally emails his Form 16 and bank statements: a 19 MB scan, from his office account at an engineering firm whose mail is hosted elsewhere. Seconds later another message arrives: the Income Tax Department has approved a refund of ₹18,450, and she should "verify her bank account" through a link. Its domain was registered two days ago, one letter off the real tax portal's name.

At 9:55 she searches twelve years of mail for last year's return of another client. Her account is at 14.6 GB of its 15 GB. And that morning her firm sent a filing reminder to 6,200 clients, about 5,400 of them on Gmail, which made it, for one day, a bulk sender. The service has to take Ravi's 19 MB from a server it has never seen, keep the fake refund out of her inbox, keep the scan for ten years, find the old PDF in a second, and decide whether her firm's own reminder deserves an inbox at all.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company that runs email for millions of businesses. The interviewer pastes the raw headers of one message into the shared document. Near the top: Received: from mail.example-engineering.in, then Authentication-Results: spf=pass dkim=pass dmarc=pass, then X-Spam-Verdict: inbox. At the bottom, an attachment of 26 MB.

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Inbound Mail#

Storage#

Reads, Search and Sending#

Key Metrics Summary#

System Architecture#

%% title: An email service: accept, queue, filter, file, serve
flowchart TD
  subgraph L1["Layer 1 · The internet"]
    direction LR
    SND@{ icon: "sd:globe", label: "Sending servers<br/>millions, not ours" }
    DNS@{ icon: "sd:dns", label: "DNS<br/>MX, SPF, DKIM, DMARC" }
  end
  subgraph L2["Layer 2 · SMTP edge"]
    direction LR
    MX@{ icon: "sd:mail", label: "MX servers<br/>TLS, IP reputation, RCPT checks" }
    LIM["Limits<br/>per IP, domain, mailbox"]:::service
  end
  subgraph L3["Layer 3 · Accept and filter"]
    direction LR
    Q@{ icon: "sd:queue", label: "Delivery queue<br/>replicated before 250 OK" }
    FILT["Spam and phishing filter<br/>auth, reputation, content, user"]:::hot
    ML@{ icon: "sd:ml", label: "Models<br/>trained on spam reports" }
  end
  subgraph L4["Layer 4 · Mailbox, partitioned by user"]
    direction LR
    MBX["Mailbox service<br/>labels, threads, quotas"]:::core
    META@{ icon: "sd:database", label: "Metadata store<br/>per user, strongly consistent" }
    IDX@{ icon: "sd:index", label: "Search index<br/>one per mailbox" }
  end
  subgraph L5["Layer 5 · Bytes, users and the way out"]
    direction LR
    BLOB@{ icon: "sd:object-storage", label: "Blob store<br/>hot replicas, cold erasure-coded" }
    BAK@{ icon: "sd:file", label: "Offline backup<br/>out of software's reach" }
    APP@{ icon: "sd:browser", label: "Web and apps" }
    OUT["Outbound MTA<br/>DKIM signing, retries"]:::service
  end
  SND -.->|"MX lookup"| DNS
  SND ==> MX
  MX --> LIM
  MX -.->|"SPF, DKIM keys"| DNS
  MX ==>|"accept"| Q
  Q ==> FILT
  ML -.-> FILT
  FILT ==> MBX
  MBX --> META
  MBX --> BLOB
  MBX -->|"new message"| IDX
  BLOB -.-> BAK
  APP ==> MBX
  APP -->|"send"| OUT
  OUT ==> SND
  class SND external
  class DNS,MX edge
  class Q queue
  class ML service
  class META,BLOB,BAK store
  class IDX cache
  class APP client

KEY INSIGHTS:

• "250 OK" is a promise. Before it, refuse cheaply; after it, deliver or report failure. The durable queue sits exactly on that line
• Partition everything by mailbox. Metadata, index, quotas and limits are per user, so no request needs two users' data
• Bytes and metadata live apart. Tiny, constantly changing labels and huge, never-changing bodies get different stores

Data Flow: Ravi's Scan Reaches Meera#

%% title: Ravi's 19 MB scan, from his office server to Meera's inbox
sequenceDiagram
  autonumber
  participant R as Ravi's mail server
  participant D as DNS
  participant M as MX edge
  participant Q as Delivery queue
  participant F as Filter
  participant B as Mailbox service
  participant I as Search index
  R->>D: MX for Meera's domain?
  D-->>R: mx edge hosts, by preference
  R->>M: EHLO, STARTTLS, MAIL FROM, RCPT TO Meera
  M->>M: IP reputation ok, mailbox exists, under limits
  R->>M: DATA, 26 MB with base64
  M->>M: SPF pass, DKIM pass
  M->>Q: Write to 3 zones, wait for 2
  Q-->>M: Durable
  M-->>R: 250 OK, queued as q7f3
  Q->>F: Message q7f3
  F->>F: DMARC pass, sender known, content clean
  F->>B: Deliver to Inbox
  B-->>I: Index new message

MEANWHILE, IN PARALLEL:

• The fake refund notice arrives on another connection. Its domain is two days old and has no history; its display name claims a government department its domain has nothing to do with. It is accepted, scored as phishing, and filed in Spam with a warning, never shown in the inbox
• Meera's phone gets a push notification for Ravi's message; the scan's text is extracted and added to her index a few seconds after the message itself

Deep Dive 1: The SMTP Front Door: Accept, Queue, Never Lose (Week 3 Concepts: Queues and Backpressure; Week 2: Idempotency)#

This builds on Week 3, Day 1 · Queue vs Stream, Week 3, Day 3 · Backpressure and Flow Control and Week 2, Day 2 · Idempotency in Practice.

What is published: RFC 5321 says a server that sends "250 OK" after DATA "MUST NOT lose the message for frivolous reasons, such as because the host later crashes or because of a predictable resource shortage", that retries "SHOULD be at least 30 minutes" apart and that "the give-up time generally needs to be at least 4-5 days". A DNS lookup of gmail.com in October 2026 listed five MX hosts at different preferences, so senders have fallbacks (our observation, not a Google statement). Gmail's sender guidelines require TLS, and tell senders rate-limited with 4.7.28 to stop for at least 10 minutes, then reconnect with one connection and add more one at a time.

%% title: One message at the edge: refuse cheaply, accept durably
flowchart TD
  C["Connection from an IP"]:::service
  REP{"IP reputation<br/>and rate ok?"}
  RCPT{"Recipient exists,<br/>mailbox under limits?"}
  DATA["Receive data<br/>size cap, SPF, DKIM"]:::service
  DUP{"Seen this<br/>message key?"}
  W@{ icon: "sd:queue", label: "Queue write<br/>3 zones, wait for 2" }
  OK(["250 OK<br/>now our responsibility"]):::ok
  T(["4xx · try later<br/>sender keeps it"]):::bad
  P(["5xx · refused<br/>sender reports it"]):::bad
  C ==> REP
  REP -->|"overloaded or throttled"| T
  REP -->|"known spam source"| P
  REP ==> RCPT
  RCPT -->|"no such user"| P
  RCPT ==> DATA ==> DUP
  DUP -->|"yes"| OK
  DUP ==>|"no"| W ==> OK
  class W queue

• Refuse at the cheapest moment: a known-bad IP at connection, an unknown recipient at RCPT TO (550 5.1.1), an oversized message when its size is declared, all before receiving megabytes or running models.
• One message, many recipients, one copy: received once, stored once, queued as one small delivery per recipient.
• Partition the queue by recipient mailbox shard, so a flood to one address backs up only its own partition.
• Backpressure through the protocol: as queue write latency rises, the edge answers 451 to a growing share of new mail, low-reputation senders first. Mail waits on the senders' disks, not ours.
• A queue, not a log: each delivery is consumed once, retried with backoff, dead-lettered with an alert if it keeps failing, and finally bounced to the sender, the "report failure" half of RFC 5321's promise.

# smtp/accept.py

"""
The end of SMTP DATA (illustrative): 250 only once durable in two zones; a retry
with the same key is acknowledged, not queued twice; under pressure, 451.
"""

import hashlib

QUORUM = 2                      # of 3 zone replicas


def message_key(headers: dict, body: bytes, rcpt: str) -> str:
    mid = headers.get("Message-ID", "")
    return hashlib.sha256(f"{mid}|{rcpt}|".encode() + hashlib.sha256(body).digest()).hexdigest()


def on_data_end(conn, headers: dict, body: bytes, rcpts: list, queue, seen, pressure: float) -> str:
    if pressure > 0.9 or (pressure > 0.7 and conn.reputation < 0.5):
        return "451 4.3.2 Try again later"                       # backpressure, sender retries
    blob_id = queue.put_blob(body, replicas=3, wait_for=QUORUM)    # written once, shared by all rcpts
    if blob_id is None:
        return "451 4.3.0 Temporary storage failure"
    for rcpt in rcpts:
        k = message_key(headers, body, rcpt)
        if seen.add_if_absent(k, ttl_days=5):                       # False if a retry already queued it
            queue.enqueue(shard_of(rcpt), {"rcpt": rcpt, "blob": blob_id, "key": k,
                                           "auth": conn.auth_results, "ip": conn.ip})
    return f"250 2.0.0 OK queued as {blob_id[:8]}"

Deep Dive 2: Spam and Phishing: Fifteen Billion No's a Day (Week 9 Concepts: Security Architecture; Week 8: Event Ingestion)#

This builds on Week 9, Day 5 · Security Architecture and Week 8, Day 1 · Event Ingestion at Scale.

What is published: in 2015 the Gmail team wrote that "Report spam" and "Not spam" clicks train the filters, that it used "an artificial neural network" for "especially sneaky spam", and that the filter could "reflect these individual preferences". In 2019 Google added TensorFlow models alongside its "existing ML and rules-based protections", blocking "around 100 million additional spam messages every day", among them image-based messages, hidden embedded content and "messages from newly created domains that try to hide a low volume of spammy messages within legitimate traffic". In 2023 Google's security blog wrote that attackers "use homoglyphs, invisible characters, and keyword stuffing", and that switching Gmail's spam classifier to the ~200,000-parameter RETVec vectorizer raised detection by 38%, cut false positives by 19.4% and cut TPU usage by 83%. Since 2013 Gmail has served images "through Google's own secure proxy servers", checked "for known viruses or malware", and it blocks executable attachments such as .exe, .js and .msi, also inside archives.

%% title: The filter funnel: cheap signals first, models on what's left
flowchart TD
  IN["Accepted message<br/>from the queue"]:::service
  AUTH["Authentication<br/>SPF, DKIM, DMARC result"]:::service
  REP@{ icon: "sd:database", label: "Reputation<br/>domain, IP, URL, age" }
  CONT["Content models<br/>text, links, attachments, images"]:::hot
  USR["Per-user signals<br/>contacts, past reports"]:::service
  V{"Score"}
  INB(["Inbox"]):::ok
  WARN(["Inbox with a warning"]):::ok
  SPAM(["Spam folder"]):::bad
  FB@{ icon: "sd:stream", label: "Report spam, Not spam<br/>events" }
  IN ==> AUTH ==> CONT
  REP -.-> AUTH
  CONT ==> USR ==> V
  V -->|"low"| INB
  V -->|"middle"| WARN
  V ==>|"high"| SPAM
  FB -.->|"minutes"| REP
  FB -.->|"retraining"| CONT
  class REP store
  class FB queue

• Order by cost. Reputation and authentication are a few cache reads; content models run only on what those can't settle. Much bulk spam never gets past the SMTP edge.
• Reputation is keyed by authenticated identity. An IP shared by thousands of senders at an email service provider says little; a domain that passed DMARC is an identity that can earn or lose standing.
• New is suspicious. A domain's age, first-seen time and volume curve are features, the case Google's 2019 post singles out.
• Read like an attacker writes. Filters that tokenise into dictionary words miss homoglyphs and invisible characters; character-level models like RETVec don't, and the 83% TPU saving shows model cost is a design constraint at this volume.
• Links and attachments are content too. Links are checked against known-bad pages, dangerous file types are refused, and images come through a proxy, so opening a message doesn't reveal the reader's IP.
• Feedback is a stream. Spam reports, aggregated per sending domain over minutes, move reputation fast; over days they become training labels; a user's own reports tune that user's verdicts.
• When unsure, deliver with a warning. Wanted mail in spam has the tighter bar (under 0.05%), so a middle band of scores gets a banner instead of being hidden.

# filter/verdict.py

"""
A filter verdict (illustrative): cheap signals settle what they can; the content
score is adjusted by reputation and the user's history; phishing has a lower bar.
"""

from dataclasses import dataclass


@dataclass
class Signals:
    dmarc: str            # "pass", "fail" or "none"
    domain_rep: float     # 0 (bad) .. 1 (good); 0.5 for unknown
    domain_age_days: int
    in_contacts: bool
    user_reported_sender: bool


def verdict(s: Signals, content_model) -> str:
    if s.domain_rep < 0.05 and s.dmarc != "pass":
        return "SPAM"                                   # settled without the model
    if s.in_contacts and s.dmarc == "pass" and s.domain_rep > 0.8:
        return "INBOX"                                  # settled without the model
    spam, phish = content_model.score()                 # the expensive step, on what's left
    prior = (0.5 - s.domain_rep) * 0.4
    if s.domain_age_days < 7:
        prior += 0.15                                   # new domains start untrusted
    if s.user_reported_sender:
        prior += 0.3                                    # this user doesn't want this sender
    if phish + prior > 0.6:
        return "SPAM_WITH_WARNING"                      # phishing: lower bar, loud warning
    if spam + prior > 0.85:
        return "SPAM"                                   # ordinary spam: higher bar
    if spam + prior > 0.6:
        return "INBOX_WITH_WARNING"
    return "INBOX"

Deep Dive 3: Who Sent This? SPF, DKIM, DMARC and the 2024 Bulk-Sender Rules (Week 9 Concepts: Security Architecture; Week 1: Rate Limiting)#

This builds on Week 9, Day 5 · Security Architecture and Week 1, Day 3 · Rate Limiting at Scale.

What is published: Gmail's sender guidelines require, from 1 February 2024, that all senders to personal Gmail accounts use SPF or DKIM, valid forward and reverse DNS (PTR), TLS, and keep the spam rate in Postmaster Tools below 0.3%. Senders of more than 5,000 messages a day must also use SPF and DKIM, publish DMARC ("Your DMARC enforcement policy can be set to none"), align From: with the SPF or DKIM domain, and support one-click unsubscribe; DKIM keys must be at least 1,024 bits. Google's announcement (October 2023) added that unsubscribes must be processed "within two days", and that after Gmail began requiring some authentication the year before, unauthenticated messages fell by 75%; it quoted Yahoo backing the same changes. The FAQ says to stay below 0.1% and never reach 0.3%: from June 2024, bulk senders above 0.3% get no mitigation until they've been below it for 7 consecutive days, and from November 2025 Gmail is "ramping up its enforcement", with "temporary and permanent rejections": the FAQ lists temporary "rate limited" codes such as 4.7.27 (SPF), 4.7.30 (DKIM), 4.7.31 (DMARC), 4.7.32 (alignment) and 4.7.29 (TLS), and permanent blocks such as 5.7.27 (SPF) and 5.7.30 (DKIM). Gmail tracks limits per domain and per IP; an IP's quota is shared by every domain sending from it.

%% title: Checking one message's sender, from envelope to policy
flowchart TD
  MSG["Incoming message<br/>MAIL FROM, From:, DKIM-Signature"]:::service
  SPF{"SPF<br/>is this IP allowed for<br/>the MAIL FROM domain?"}
  DK{"DKIM<br/>does the signature verify<br/>with the d= domain's key?"}
  AL{"DMARC alignment<br/>does either domain match<br/>the From: domain?"}
  POL@{ icon: "sd:dns", label: "DMARC record<br/>p=none, quarantine or reject" }
  REPD@{ icon: "sd:database", label: "Reputation<br/>per authenticated domain" }
  PASS(["DMARC pass<br/>identity to judge"]):::ok
  FAIL(["DMARC fail<br/>apply the domain's policy"]):::bad
  MSG ==> SPF
  MSG ==> DK
  SPF --> AL
  DK --> AL
  AL ==>|"aligned pass"| PASS ==> REPD
  AL -->|"no aligned pass"| FAIL
  POL -.-> FAIL
  class REPD store

• SPF (RFC 7208) checks the envelope sender's domain, which humans never see, against a DNS list of allowed IPs, in at most 10 DNS-querying lookups. It breaks when mail is forwarded.
• DKIM (RFC 6376) signs headers and body with a private key whose public half is in DNS under the d= domain. It survives forwarding, but breaks if a mailing list edits the subject or adds a footer.
• DMARC (RFC 7489) is what makes the visible From: trustworthy: pass means SPF or DKIM passed for a domain aligned with From:. The domain owner publishes a policy, none, quarantine or reject, and an address for aggregate reports, so it can see who is sending as its domain. A DNS lookup of _dmarc.gmail.com in October 2026 returned p=none; sp=quarantine with a reporting address; Gmail's guidelines say it "will begin using a DMARC quarantine enforcement policy" and warn senders not to impersonate Gmail From: headers.
• Reputation attaches to the aligned domain. Once a message passes DMARC, per-domain counters (volume, spam reports, unsubscribes) belong to someone. That's why the 2024 rules matter beyond spam: every bulk sender gets an identity that can be held to account.
• A bulk-sender counter per domain, approximate and merged across regions every few minutes, finds senders crossing 5,000 a day.
• Rate limits per domain and per IP. A sender over what its reputation allows gets 4.7.28, and SMTP's retries spread its mail out; Gmail tells senders to pause at least 10 minutes, then add connections one at a time.
• Our outbound side follows the same rules: DKIM-signed with the user's domain, over TLS, retried per RFC 5321, with bounces processed so our own IPs keep their reputation at other providers.

# auth/dmarc.py

"""
DMARC for one message (illustrative, after RFC 7489): pass if SPF or DKIM passed
for a domain aligned with From:; relaxed alignment compares organisational domains.
"""


def org_domain(d: str, public_suffixes: set) -> str:
    labels = d.lower().rstrip(".").split(".")
    for i in range(len(labels)):
        if ".".join(labels[i + 1:]) in public_suffixes:
            return ".".join(labels[i:])                 # e.g. mail.example.co.in -> example.co.in
    return d.lower()


def aligned(a: str, b: str, mode: str, psl: set) -> bool:
    return a.lower() == b.lower() if mode == "s" else org_domain(a, psl) == org_domain(b, psl)


def dmarc(from_domain: str, spf: tuple, dkim: list, record: dict, psl: set) -> dict:
    """spf = (result, mail_from_domain); dkim = [(result, d_domain), ...]; record from DNS or {}."""
    spf_ok = spf[0] == "pass" and aligned(spf[1], from_domain, record.get("aspf", "r"), psl)
    dkim_ok = any(r == "pass" and aligned(d, from_domain, record.get("adkim", "r"), psl) for r, d in dkim)
    if spf_ok or dkim_ok:
        return {"result": "pass", "identity": org_domain(from_domain, psl)}
    if not record:
        return {"result": "none", "identity": None}     # no policy: judged on other signals
    return {"result": "fail", "policy": record.get("p", "none"), "identity": None}

Deep Dive 4: Storing Mail Cheaply but Durably (Week 1 Concepts: Replication Trade-offs; Week 4: Multi-Tier; Week 10: Deployment Strategies)#

This builds on Week 1, Day 2 · Replication Trade-offs, Week 4, Day 5 · Multi-Tier Caching and Week 10, Day 3 · Deployment Strategies. The 2011 incident is the war story A software bug deleted Gmail mail from several copies at once.

What is published: in February 2011 Ben Treynor, Google's VP of Engineering, wrote that 0.02% of Gmail users had found their accounts empty: Google kept "multiple copies of your data, in multiple data centers", but "a storage software update" had a bug that hit several copies. Google stopped the rollout and reverted, and restored the mail from tape: "Since the tapes are offline, they're protected from such software bugs", though restoring took "hours ... instead of milliseconds". Mail sent to affected users in a 20-hour window "was likely not delivered", and its senders were notified. Google Cloud wrote in 2021 that Colossus underpins Gmail, and that Colossus in general keeps "the hottest data" on flash and moves data to "larger capacity drives as it ages and becomes colder"; and in 2023 that Spanner supports Gmail. Gmail keeps deleted messages in Trash for 30 days. How Gmail lays out mailboxes on these systems isn't published.

%% title: One message's bytes, from hot to cold, with a copy software can't touch
flowchart TD
  NEW["New message<br/>body and attachments"]:::service
  CA["Content address<br/>hash of the bytes, refcount"]:::core
  HOT@{ icon: "sd:object-storage", label: "Hot tier · flash and disk<br/>3 replicas in 3 zones" }
  COLD@{ icon: "sd:object-storage", label: "Cold tier · large drives<br/>erasure-coded 6 + 3" }
  META@{ icon: "sd:database", label: "Mailbox metadata<br/>labels, thread, blob refs" }
  DEL["Soft delete<br/>tombstone, purge after days"]:::service
  BAK@{ icon: "sd:file", label: "Offline backup<br/>different software, not online" }
  NEW ==> CA ==> HOT
  HOT ==>|"after ~30 days"| COLD
  CA -.-> META
  META --> DEL
  HOT -.->|"daily, incremental"| BAK
  COLD -.-> BAK
  class HOT,COLD,META,BAK store

• Metadata in a strongly consistent store, partitioned by user. Labels, read state and blob references are small and change all day; one key range per user makes "archive this thread" one transaction on one shard.
• Bodies and attachments as immutable blobs, addressed by hash. Ravi's scan is stored once; a newsletter's logo sent to 3 million users is stored once, with a reference count. A blob is never edited, only referenced or released, which keeps replication and erasure coding simple.
• Tier by age. Replicas on fast media for about 30 days, then erasure-coded fragments on large drives: ~5.9 EB of raw disk instead of ~11 EB on the Phase 2 estimate.
• Deletes are slow on purpose. Trash for 30 days, then the metadata row goes and the blob's count drops; a blob at zero is purged only after a further grace period, so a buggy mass delete can be undone from tombstones.
• A copy software can't reach. Backups on media the online storage software never mounts, written by a separate, simpler program and verified by reading back. They're slow to restore, so they're the last line, not the first.
• Roll out storage software one cell at a time: canary, 1%, 10%, each step gated on blob checksums, reference counts and "mailboxes that shrank by more than half".

# storage/blobs.py

"""
Content-addressed blobs with reference counts and slow purges (illustrative): a blob
is purged only after a grace period with no references, so a mass delete is reversible.
"""

import hashlib
import time

GRACE_S = 14 * 86400


def put(store, data: bytes) -> str:
    h = hashlib.sha256(data).hexdigest()
    if not store.exists(h):
        store.write(h, data, replicas=3, zones=3, tier="hot")
    store.incr_ref(h)
    return h


def release(store, h: str) -> None:
    if store.decr_ref(h) == 0:
        store.mark_unreferenced(h, at=time.time())        # not deleted yet


def purge_sweep(store, max_per_run: int, deleting_rate_alarm) -> int:
    if deleting_rate_alarm():                              # far more unreferenced than usual?
        return 0                                           # stop: maybe a bug, not users
    n = 0
    for h, since in store.unreferenced(older_than=time.time() - GRACE_S, limit=max_per_run):
        if store.ref(h) == 0:
            store.delete(h)
            n += 1
    return n


def cool(store, h: str) -> None:
    store.reencode(h, scheme=(6, 3), tier="cold")         # 1.5x instead of 3x, once rarely read

Deep Dive 5: Search Across a Mailbox (Week 7 Concepts: Search Fundamentals, Indexing Pipeline)#

This builds on Week 7, Day 1 · Search Fundamentals and Architecture, Week 7, Day 2 · Indexing Pipeline and Week 7, Day 5 · Search Operations and Scale.

What is published: Google's 2004 announcement called Gmail "a free search-based webmail service", "built on Google search technology", to "quickly search every email they've ever sent or received". Gmail's help lists the operators: from:, to:, subject:, after:/before:, older_than:, has:attachment, label:, in:anywhere (Spam and Trash too), larger:/smaller:, AROUND for words near each other, rfc822msgid:, and OR, - and { }. Operators match messages, then whole conversations are shown. How Gmail's index is built isn't published.

%% title: One mailbox's index, built from the delivery stream
flowchart TD
  DLV@{ icon: "sd:stream", label: "Delivery events<br/>new, deleted, moved" }
  TOK["Tokenise<br/>fields, words with positions"]:::service
  EXT["Attachment text<br/>PDF, docs, async"]:::service
  MEM["Fresh segment<br/>in memory, seconds old"]:::hot
  SEG@{ icon: "sd:index", label: "On-disk segments<br/>merged as they grow" }
  BM@{ icon: "sd:database", label: "Label bitmaps<br/>in mailbox metadata" }
  Q["Query<br/>parse operators, plan by rarity"]:::core
  RES(["Message ids<br/>grouped into threads, newest first"]):::ok
  DLV ==> TOK ==> MEM
  DLV -.-> EXT -.-> MEM
  MEM -->|"flush and merge"| SEG
  SEG ==> Q
  MEM ==> Q
  BM -.->|"label: filter"| Q
  Q ==> RES
  class DLV queue
  class SEG cache
  class BM store

• Partition the index like the mailbox. It lives on the user's shard: one shard per query, no fan-out, and the access check is "is this your mailbox". At ~66 MB (a twelve-year mailbox, a few hundred), hot parts fit in memory while the user is active.
• Fields as columns, words as postings. Sender, date, size and flags are per-message columns (larger:10M scans one); words map to message ids with positions, which AROUND and phrases need.
• Log-structured. New mail goes into a small segment; segments are merged in the background; deletes are a bitmap applied at query time until the next merge.
• Index from the delivery stream, not the SMTP path, so a slow indexer delays search by seconds but never delays delivery.
• Sort by date by default: mail search is mostly "find the one I know exists". Spam and Trash are one more bitmap, included only for in:anywhere.

# search/query.py

"""
A Gmail-style query over one mailbox's index (illustrative): each term becomes a set
of message ids, intersected smallest first; labels come from metadata bitmaps.
"""

import re
import time

TOKEN = re.compile(r'(-?)(\w+):(\S+)|(-?)"([^"]+)"|(-?)(\S+)')


def parse(q: str) -> list:
    terms = []
    for m in TOKEN.finditer(q):
        if m.group(2):
            terms.append((m.group(1) == "-", m.group(2).lower(), m.group(3).lower()))
        elif m.group(5):
            terms.append((m.group(4) == "-", "phrase", m.group(5).lower()))
        else:
            terms.append((m.group(6) == "-", "word", m.group(7).lower()))
    return terms


def ids_for(idx, meta, kind: str, value: str) -> set:
    if kind == "from":
        return idx.field("from", value)
    if kind == "has" and value == "attachment":
        return idx.flag("attachment")
    if kind == "older_than":
        n, unit = int(value[:-1]), value[-1]
        return idx.date_before(time.time() - n * {"d": 86400, "m": 30 * 86400, "y": 365 * 86400}[unit])
    if kind == "label":
        return meta.label_bitmap(value)
    if kind == "phrase":
        return idx.phrase(value.split())                  # positions must be consecutive
    return idx.postings(value)                            # a plain word


def search(idx, meta, q: str, limit: int = 50) -> list:
    pos, neg = [], []
    for negate, kind, value in parse(q):
        (neg if negate else pos).append(ids_for(idx, meta, kind, value))
    if not pos:
        return []
    pos.sort(key=len)                                     # rarest first keeps every step small
    hits = set(pos[0])
    for s in pos[1:]:
        hits &= s
    for s in neg:
        hits -= s
    hits -= meta.in_spam_or_trash()                       # unless in:anywhere was asked for
    return sorted(hits, key=idx.date, reverse=True)[:limit]

Deep Dive 6: Quotas and Rate Limits (Week 1 Concepts: Rate Limiting at Scale; Week 9: Noisy Neighbours)#

This builds on Week 1, Day 3 · Rate Limiting at Scale and Week 9, Day 2 · Noisy Neighbor Prevention.

What is published: each Google Account has up to 15 GB shared across Gmail, Drive and Photos; over quota, "your ability to send and receive email in Gmail can be impacted", and after 2 years over quota content "may be removed", with at least 3 months' notice; after a big deletion, usage "normally takes up to 48–72 hours" to update (Google One Help). Personal accounts unused for 2 years may be deleted, from December 2023 at the earliest (Google, May 2023). Personal Gmail's sending limit is around 500 recipients in one email or 500 emails a day (Gmail Help). A Workspace account may send 2,000 messages and 10,000 recipients a day "over a rolling 24-hour period", and receive 60 messages a minute, 3,600 an hour and 86,400 a day, each up to 50 MB (70 MB on Enterprise Plus) after encoding, which "adds about a 37% increase". At the receiving limit, new messages bounce for "about 24 hours" and "can't be recovered" (Google Workspace Admin Help).

%% title: Four kinds of limit, checked at different moments
flowchart TD
  IN@{ icon: "sd:mail", label: "Inbound message<br/>from a sending IP and domain" }
  SL["Sender limits<br/>per IP, per domain, by reputation"]:::service
  RL["Receiving limit<br/>per mailbox: minute, hour, day"]:::hot
  SQ["Storage quota<br/>cached usage plus reservation"]:::core
  OUTM@{ icon: "sd:user", label: "User sending<br/>web, app, SMTP, API" }
  OL["Sending limits<br/>per account, rolling 24 h"]:::service
  OK(["Accepted or sent"]):::ok
  T(["4xx · try later<br/>421, 451, 452"]):::bad
  IN ==> SL ==> RL ==> SQ ==> OK
  SL -->|"over"| T
  RL -->|"over"| T
  SQ -->|"full"| T
  OUTM ==> OL ==> OK
  OL -->|"over"| T
  class IN external
  class OUTM client

• Three windows, one rate. 60 a minute, 3,600 an hour and 86,400 a day are all one message a second: checking all three allows a burst of 60 but no sustained flood. Sliding-window counters on the mailbox's own shard make each check a few increments.
• Storage quota: a fast estimate and a slow truth. Exact usage is recomputed asynchronously (hence Google's 48 to 72 hours); the delivery path reads a cached counter and reserves the message's size. Being off by a few hundred megabytes is fine; reading every blob per delivery is not.
• Temporary when the user can fix it. "Mailbox full" and "receiving too fast" are 4xx, so the sender's queue holds the message; 5xx is for things that won't change, like a mailbox that doesn't exist.
• Sending limits stop compromised accounts, whose first use is usually spam: a rolling 24-hour cap, stricter for new accounts, protects our outbound IPs' reputation.
• Guard the automation. In December 2020 an old quota system reported Google's account service as using nothing, automation cut its quota, and sign-ins failed for 47 minutes (war story: Google's sign-in went down because a quota system thought it was using nothing). Treat "usage dropped to zero" as a likely bug, and cap how fast a limit can tighten.

# limits/receiving.py

"""
A per-mailbox receiving limit (illustrative) at Workspace's published windows: 60 a
minute, 3,600 an hour, 86,400 a day, each a sliding-window counter.
"""

import time

WINDOWS = ((60, 60), (3600, 3600), (86400, 86400))      # (seconds, max messages)


def sliding_count(kv, key: str, window_s: int, now: float) -> float:
    cur_start = int(now // window_s) * window_s
    cur = kv.get(f"{key}:{window_s}:{cur_start}") or 0
    prev = kv.get(f"{key}:{window_s}:{cur_start - window_s}") or 0
    elapsed = (now - cur_start) / window_s
    return cur + prev * (1 - elapsed)


def admit(kv, mailbox: str, now: float = None) -> str:
    now = now or time.time()
    key = f"rcv:{mailbox}"
    for window_s, limit in WINDOWS:
        if sliding_count(kv, key, window_s, now) + 1 > limit:
            return "450 4.2.1 Mailbox receiving too fast, try later"
    for window_s, _ in WINDOWS:
        b = int(now // window_s) * window_s
        kv.incr(f"{key}:{window_s}:{b}", ttl_s=2 * window_s)
    return "OK"


def reserve_storage(quota, mailbox: str, size: int) -> str:
    used, limit = quota.cached(mailbox)                   # fast estimate, reconciled later
    if used + quota.reserved(mailbox) + size > limit:
        return "452 4.2.2 Mailbox full"                   # temporary: the sender keeps retrying
    quota.reserve(mailbox, size, ttl_s=600)               # released or committed at write time
    return "OK"

Meera's Evening#

Edge Cases#

Monitoring Dashboard#

SLOs for the Platform#

The interviewer leans back. "One more."

Summary: Concepts Applied from 10-Week Course#

Why Gmail Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Receiving:

Filtering and senders:

Storage, search and limits:

Sources#

Google and Gmail:

• Blog

  Neil Kumaran — New Gmail protections for a safer, less spammy inbox (nearly 15 billion unwanted emails blocked a day; 99.9%+; unauthenticated mail down 75%; the February 2024 bulk-sender rules; unsubscribes within two days; Yahoo's backing), Google, 3 October 2023: https://blog.google/products/gmail/gmail-security-authentication-spam-protection/

• Blog

  Neil Kumaran — Spam does not bring us joy: ridding Gmail of 100 million more spam messages with TensorFlow (1.5 billion monthly users; 5 million paying businesses; ~100 million more spam messages blocked a day; new domains), Google Workspace blog, 7 February 2019: https://workspace.google.com/blog/product-announcements/ridding-gmail-of-100-million-more-spam-messages-with-tensorflow

• Blog

  Sri Harsha Somanchi — The mail you want, not the spam you don't (under 0.1% spam in the inbox; under 0.05% wanted mail in spam; a neural network; per-user preferences), Official Gmail Blog, 9 July 2015: https://gmail.googleblog.com/2015/07/the-mail-you-want-not-spam-you-dont.html

• Blog

  Elie Bursztein and Marina Zhang — Improving Text Classification Resilience and Efficiency with RETVec (detection +38%, false positives −19.4%, TPU use −83%; ~200k parameters), Google Security Blog, 29 November 2023: https://security.googleblog.com/2023/11/improving-text-classification.html

• Blog

  Ben Treynor — Gmail back soon for everyone (0.02% of users; a storage software bug hit several copies; restored from offline tape; undelivered mail window), Official Gmail Blog, 28 February to 2 March 2011: https://gmail.googleblog.com/2011/02/gmail-back-soon-for-everyone.html

• Blog

  John Rae-Grant — Images now showing (images served through Google's secure proxy servers), Official Gmail Blog, 12 December 2013: https://gmail.googleblog.com/2013/12/images-now-showing.html

• Blog

  Google — Google Gets the Message, Launches Gmail (search-based webmail; 1 GB free), 1 April 2004: https://googlepress.blogspot.com/2004/04/google-gets-message-launches-gmail.html

• Blog

  Ruth Kricheli — Updating our inactive account policies (2 years; from December 2023), Google, 16 May 2023: https://blog.google/technology/safety-security/updating-our-inactive-account-policies/

• Blog

  Dean Hildebrand and Denis Serenyi — A peek behind Colossus, Google's file system (underpins Gmail; hot data on flash, colder data on larger drives), Google Cloud blog, 20 April 2021: https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system

• Blog

  Jagdeep Singh and Pritam Shah — Cloud Spanner is now half the cost of Amazon DynamoDB (Spanner supports Ads, Gmail and Photos), Google Cloud blog, 11 October 2023: https://cloud.google.com/blog/products/databases/announcing-cloud-spanner-price-performance-updates

Official help pages:

• Docs

  Gmail Help — Email sender guidelines (all senders and 5,000+ a day; spam rate; DKIM key length; one-click unsubscribe; 4.7.28 back-off): https://support.google.com/a/answer/81126

• Docs

  Gmail Help — Email sender guidelines FAQ (enforcement; error codes; mitigation; November 2025 ramp-up): https://support.google.com/a/answer/14229414

• Read

  Gmail Help — Limits for sending and getting mail: https://support.google.com/mail/answer/22839

• Read

  Google Workspace Admin Help — Gmail sending limits in Google Workspace: https://support.google.com/a/answer/166852

• Read

  Google Workspace Admin Help — Gmail receiving limits in Google Workspace: https://support.google.com/a/answer/1366776

• Read

  Gmail Help — Attachment size limits: https://support.google.com/mail/answer/6584

• Read

  Gmail Help — File types blocked in Gmail: https://support.google.com/mail/answer/6590

• Read

  Gmail Help — Refine searches in Gmail: https://support.google.com/mail/answer/7190

• Read

  Gmail Help — Delete messages in Gmail (Trash, 30 days): https://support.google.com/mail/answer/7401

• Read

  Google One Help — How your Google storage works (15 GB; over quota; 48 to 72 hours): https://support.google.com/googleone/answer/9312312

Standards:

• Docs

  RFC 5321 (SMTP), RFC 7208 (SPF), RFC 6376 (DKIM), RFC 7489 (DMARC), RFC 8058 (one-click unsubscribe), RFC 8617 (ARC), RFC 3463 (status codes, X.2.2 mailbox full): https://www.rfc-editor.org/

Reporting and observation:

• Blog

  Android Police — Gmail now has more than 1.5 billion active users (Gmail's post of 26 October 2018), 26 October 2018: https://www.androidpolice.com/2018/10/26/gmail-now-1-5-billion-active-users/

• Read

  DNS lookups of gmail.com (five MX hosts) and _dmarc.gmail.com (p=none; sp=quarantine), 28 October 2026

Design note: the architecture, code, thresholds, dashboards and every number labelled as an assumption or estimate are an illustrative design for the problem, not a description of Gmail's internal systems beyond what Google has published. Messages per mailbox, message sizes, storage per mailbox, daily users, read and search rates, the deduplication saving, the erasure-coding scheme and the final challenge's figures are assumptions. Meera, Ravi and their messages are fictional.

Further Reading#

Related systems to study:

• Read

  WhatsApp (Bonus Problem 2 in this series): messages that are deleted once delivered, the opposite storage choice to email's "keep forever"

• Book

  Google Search (Bonus Problem 5): one global index sharded by document, where this quest keeps a billion small ones

• Read

  AWS S3 (Bonus Problem 8): erasure coding, durability and the cost of nines

• Docs

  Google Docs (Bonus Problem 20): another Google product where every request belongs to one document or user

Foundational concepts (this course):

• Read

  Week 1, Day 2 — Replication Trade-offs

• Read

  Week 1, Day 3 — Rate Limiting at Scale

• Read

  Week 2, Day 2 — Idempotency in Practice

• Read

  Week 3, Day 1 — Queue vs Stream

• Read

  Week 3, Day 3 — Backpressure and Flow Control

• Read

  Week 4, Day 5 — Multi-Tier Caching

• Read

  Week 7, Day 1 — Search Fundamentals and Architecture

• Read

  Week 7, Day 2 — Indexing Pipeline

• Read

  Week 7, Day 5 — Search Operations and Scale

• Read

  Week 8, Day 1 — Event Ingestion at Scale

• Read

  Week 9, Day 2 — Noisy Neighbor Prevention

• Read

  Week 9, Day 5 — Security Architecture

• Read

  Week 10, Day 3 — Deployment Strategies

"Say 250 only when it's safe, judge senders by who they provably are, and keep one copy your own code can't reach."