Read the originals.

What

A Google engineer's account of how design docs are written and reviewed at Google, and what goes in one.

Key idea

A design doc records context, goals and non-goals, the chosen design and the alternatives rejected, so trade-offs are argued before code exists.

Why here

It is the written form of the HLD-first habit this part teaches: requirements and non-goals up front, boxes and arrows next, details only where they matter.

What

A classic paper from the Windows Live Services team listing practices for running large online services.

Key idea

Design for failure, automate everything, keep things simple and make operations a design input rather than an afterthought.

Why here

Its checklist reads like the non-functional requirements reference in this part: availability, scalability, operability and cost decide the architecture.

What

The reference site for C4, a lightweight notation for drawing software architecture at four zoom levels.

Key idea

Draw context, containers, components and code as separate diagrams, each for a different audience, instead of one diagram that tries to show everything.

Why here

It gives the diagram types in this part a shared vocabulary: the architecture diagram is a container view, and you zoom in only for the deep dive.

What

A chapter of Google's Site Reliability Engineering book on how user traffic reaches the right datacenter.

Key idea

DNS load balancing picks a region but caches and resolvers blur it, so a virtual IP and network load balancer spread traffic inside the datacenter.

Why here

It shows DNS and the load balancer, two building blocks from this part, working as one layered traffic-routing system.

What

Shopify explains why it kept its Rails monolith and restructured it instead of splitting into microservices.

Key idea

A modular monolith with enforced component boundaries keeps one deployable while getting much of the isolation teams want from services.

Why here

A real counterweight to 'microservices by default' in the monolith vs microservices section: the boundary matters more than the network hop.

What

Segment's story of splitting its event-delivery pipeline into one service per destination and then merging them back.

Key idea

Per-destination services multiplied queues, shared-library drift and on-call load, and one well-tested service turned out simpler and faster.

Why here

Shows the operational cost side of the microservices trade-off taught here, with the numbers that pushed a team back to a monolith.

What

A worked design exercise from Google's SRE workbook that sizes a real system step by step.

Key idea

Start with a design that works on one machine, then estimate QPS, storage, bandwidth and machine counts to see where it breaks and what it costs.

Why here

It is the estimation method of this part done in full: traffic, storage, bandwidth and infrastructure estimates driving each design decision.

What

Slides from Jeff Dean's keynote on how Google builds large systems, including the original 'numbers everyone should know'.

Key idea

Knowing rough costs (memory reference, disk seek, datacenter round trip) lets you estimate a design's performance before building it.

Why here

The source of the latency table this part teaches, plus examples of using those numbers to choose between designs.

What

An open repository of measured throughput and latency numbers for memory, disks, networks and databases, with estimation techniques.

Key idea

Benchmarks on current hardware give base rates you can multiply out to check whether a design's numbers are plausible.

Why here

An up-to-date companion to the throughput reference numbers and estimation templates in this part.

What

A free chapter of Ilya Grigorik's book on how TCP actually behaves on real networks.

Key idea

The three-way handshake, slow start and head-of-line blocking mean latency, not bandwidth, usually limits performance.

Why here

Goes deeper into the TCP chapter here and explains why connection reuse and fewer round trips matter in latency budgets.

What

Cloudflare's announcement of HTTP/3 support, with the history from HTTP/1.1 through HTTP/2 to QUIC.

Key idea

HTTP/3 runs over QUIC on UDP, combining transport and TLS handshakes and removing TCP's head-of-line blocking.

Why here

Ties together the TCP, UDP, HTTP and TLS chapters of this part: each HTTP version is a fix for the previous layer's latency cost.

What

How Slack delivers messages to millions of connected clients in real time.

Key idea

Clients hold WebSocket connections to gateway servers, and channel servers fan each message out to the gateways of every subscribed user.

Why here

A production view of the WebSocket chapter: long-lived connections, stateful gateways and pub/sub fan-out behind them.

What

Netflix's checklist of ten standard Linux commands to run in the first minute on a slow server.

Key idea

uptime, vmstat, mpstat, iostat, free, sar and top together show CPU saturation, memory pressure, disk and network I/O at a glance.

Why here

Every command maps to a concept in this part (run queues, context switches, page cache, I/O wait), so it turns theory into a debugging routine.

What

An experiment in receiving 1M UDP packets per second on Linux, step by step.

Key idea

Throughput is bounded by how the kernel spreads packets across RX queues, CPUs and sockets; SO_REUSEPORT and NUMA placement remove the bottleneck.

Why here

Shows sockets, file descriptors, CPU cores and kernel queues (all from this part) deciding real network throughput.

What

Discord's account of rewriting its Read States service from Go to Rust.

Key idea

Go's garbage collector caused latency spikes every two minutes on a large LRU cache; Rust's ownership model frees memory without GC pauses.

Why here

A concrete case of memory management and runtime behaviour, from the OS concepts here, turning into user-visible tail latency.

What

The CAP theorem's author revisits it twelve years on.

Key idea

'Pick two of three' misleads: partitions are rare, so design for detecting them, operating in a degraded mode and recovering consistency afterwards.

Why here

The authoritative reading of the CAP chapter here, and the bridge to PACELC: the real trade-off is what you give up while partitioned.

What

The paper that introduced PACELC as a refinement of CAP.

Key idea

If there is a Partition choose Availability or Consistency, Else choose Latency or Consistency; most systems pay the latency trade-off all the time.

Why here

The original source for the PACELC chapter, with Dynamo, Cassandra, PNUTS and others classified.

What

A critique of labelling real databases as CP or AP.

Key idea

CAP's definitions are narrow (linearizability, total availability), so most systems are neither; talk about specific consistency models and latency instead.

Why here

Sharpens the CAP and consistency-model chapters: in interviews, name the guarantee (linearizable, causal, read-your-writes), not the letter pair.

What

How Notion split its single Postgres database into many shards in 2021.

Key idea

They partitioned by workspace ID so a workspace's blocks, discussions and comments stay together, and ran many logical shards on fewer physical hosts.

Why here

A real partition-key decision from this lesson: pick the key your queries already scope by, keep related rows co-located, and plan the cut-over.

What

Figma's nine-month project to horizontally shard its Postgres stack after the database grew almost 100x since 2020.

Key idea

Vertical partitioning by table bought runway first; horizontal sharding then used logical shards and a query-routing proxy so data could move without app rewrites.

Why here

Walks through the vertical vs horizontal partitioning choice and the resharding problem from this lesson on a live product.

What

Discord's move of message storage from MongoDB to Cassandra.

Key idea

Messages are partitioned by channel plus a time bucket, so a busy channel's history spreads over bounded partitions and reads hit one partition at a time.

Why here

A composite partition key in practice: the key matches the main query (recent messages in a channel) and the bucket caps partition size.

What

GitHub's postmortem of 24 hours and 11 minutes of degraded service in October 2018.

Key idea

A 43-second network partition made automated failover promote MySQL primaries in another region; writes landed on both sides and had to be reconciled by hand.

Why here

The failover chapter's warning made real: automatic promotion across regions plus replication lag can cost far more than the outage it reacts to.

What

Uber's explanation of the storage and replication limits it hit with Postgres.

Key idea

Postgres replicated physical WAL changes, so write amplification on indexes multiplied replication traffic; MySQL's logical replication suited their workload better.

Why here

Makes the replication internals chapter concrete: what is shipped to replicas (physical pages vs logical rows) changes lag, bandwidth and upgrades.

What

The four limiters Stripe runs in front of its API, with sample code.

Key idea

Per-user request-rate and concurrency limiters (token buckets in Redis) sit alongside load shedders that drop low-priority traffic when the whole fleet is stressed.

Why here

Shows the distinction this lesson makes between rate limiting a client and load shedding for the system, and how to roll limiters out safely.

What

How Cloudflare counts requests for rate limiting across its global edge.

Key idea

A sliding window approximated from the current and previous fixed-window counters needs only GET, SET and INCR, yet stays very close to an exact count.

Why here

The sliding window counter algorithm from this lesson, with the memory and accuracy reasoning behind choosing it at the edge.

What

Figma compares token bucket, fixed window and sliding window log limiters before building its own.

Key idea

Its final design counts requests in small fixed windows, each 1/60th of the limit window, and sums them: a sliding window counter that is accurate but cheap in Redis memory.

Why here

A side-by-side comparison of the algorithms in this lesson, judged on the same two axes the lesson uses: accuracy and memory.

What

Discord's migration from Cassandra to ScyllaDB and the services it built in front of the database.

Key idea

Hot partitions in busy channels slowed the whole cluster; Rust data services coalesce identical concurrent reads so one query serves many requests.

Why here

A textbook hot-key story: detection by latency, mitigation by request coalescing and routing, matching the mitigation strategies in this lesson.

What

AWS explains how DynamoDB handles tables whose traffic is not spread evenly across partitions.

Key idea

Throughput used to be split equally per partition, so one hot partition was throttled; adaptive capacity now boosts hot partitions from the table's unused capacity.

Why here

Shows skew at the storage layer and a platform-level mitigation, a useful contrast with the application-level fixes in this lesson.

What

The 2022 paper describing ten years of DynamoDB's architecture.

Key idea

Partitions split by size and by consumption, and global admission control replaced fixed per-partition limits, so uneven traffic no longer means throttling.

Why here

Goes deeper on splitting hot partitions and admission control, the storage-side answers to the skew this lesson measures.

What

The official guide to running Redis Cluster.

Key idea

Keys map to 16,384 hash slots spread over primaries with replicas; clients follow MOVED/ASK redirects while slots migrate during resharding.

Why here

The mechanics behind the Redis Cluster option compared in this lesson: how sessions are sharded, replicated and failed over.

What

How Netflix replicates its EVCache data across AWS regions.

Key idea

Writes in one region are sent through a replication queue and relay to the other regions, so a member failed over to another region still finds their data.

Why here

The multi-datacenter chapter of session design: asynchronous cross-region replication of a fast key-value store, and the consistency it trades away.

What

An introduction to Ringpop, Uber's library for making stateful services horizontally scalable.

Key idea

SWIM gossip tracks membership, a consistent hash ring assigns each key an owner, and requests are forwarded to that owner.

Why here

Sticky routing done in the application tier: every request for a key reaches the node holding its state, the idea behind session affinity.

What

Pinterest's design for sharding MySQL across thousands of logical databases.

Key idea

Every object gets a 64-bit ID that embeds its shard ID, type and local ID, so any ID can be routed to its shard with no lookup service.

Why here

The capstone's short-code and sharding deep dives in one design: IDs that encode their location make reads one hop.

What

Discord's account of hot partitions and the data services it put in front of its database.

Key idea

Coalescing identical in-flight reads and routing a key's requests to one service instance stops a viral object from overloading storage.

Why here

Directly reusable for the capstone's hot-URL deep dive: a viral short link is the same hot-key problem as a busy channel.

What

Amazon's guidance on setting timeouts and retrying safely between services.

Key idea

Pick timeouts from measured downstream latency percentiles, cap retries (a retry budget or token bucket) and add jitter so retries do not synchronise.

Why here

The core advice of this lesson in Amazon's words: a timeout is a latency-percentile decision, and unbounded retries turn a slow service into an outage.

What

Google's paper on why rare slow responses dominate latency in large fan-out systems.

Key idea

When a request touches many servers, the slowest one sets the response time; hedged and tied requests cut the tail without making each server faster.

Why here

Explains why this lesson budgets for p99 rather than averages, and gives hedging as a tool alongside timeouts.

What

The SRE book chapter on how overload in one service spreads to others.

Key idea

Queues, retries and missing deadlines amplify load; propagate deadlines, shed load early and cap retries per request so failure stays local.

Why here

The cascade-failure chapter of this lesson at Google scale, including deadline propagation across a call chain.

What

Stripe explains how its API lets clients retry payments without charging twice.

Key idea

The client sends an Idempotency-Key header; the server stores the first result under that key and replays it for retries, while clients back off with jitter.

Why here

The idempotency key pattern exactly as taught in this lesson, from the company that made it standard for payment APIs.

What

A detailed implementation of idempotency keys on Postgres, with code.

Key idea

Split a request into atomic phases with recovery points stored on the key's row, so a retry resumes where the last attempt stopped, even after a crash mid-way.

Why here

Goes past the basic pattern into the edge cases this lesson warns about: partial failures between the database write and the external call.

What

How Airbnb's payments services avoid charging guests twice.

Key idea

Their Orpheus library splits each request into Pre-RPC, RPC and Post-RPC phases, keeping network calls out of database transactions and recording progress per key.

Why here

Applies this lesson's 'network timeout' problem to money: when a call times out you must know whether it happened before you retry.

What

Netflix's 2012 description of how its API isolates failing dependencies, the design behind Hystrix.

Key idea

Wrap each dependency call with a timeout, a bounded thread pool (bulkhead), a circuit breaker and a fallback, so one bad dependency cannot exhaust the server.

Why here

The origin of the circuit breaker plus bulkhead combination this lesson teaches, with fallbacks as part of the design.

What

Shopify models how its Semian circuit breaker behaves while a dependency is down.

Key idea

Thresholds, error timeout and half-open settings decide worker utilisation during an outage; badly chosen values leave a service as slow as having no breaker.

Why here

The 'when circuit breakers cause harm' chapter with numbers: configure a breaker by modelling capacity during failure, not by guessing.

What

Dream11 built a circuit-breaker plugin for its Kong API gateway, which takes up to 80 million requests a minute during the IPL.

Key idea

Breakers track failure percentage per route over a time window, fail fast when open and let a few requests through in half-open, so one failing route does not stop a whole service.

Why here

The closed, open and half-open states from this lesson at the gateway, and why breaking per route beats breaking per service.

What

Stripe's guide to receiving and verifying webhook events.

Key idea

Events are retried with exponential backoff for up to three days in live mode, may arrive out of order or twice, and carry a signature to verify against the raw body.

Why here

The sender's contract this lesson designs: at-least-once delivery, signed payloads and receivers that dedupe by event ID.

What

Razorpay's rules for how its webhooks are delivered and how merchants should handle them.

Key idea

Any non-2xx response is retried with exponential backoff for 24 hours; after that the webhook is disabled and the merchant alerted, and duplicates must be handled.

Why here

A concrete retry window and disable policy, exactly the decisions behind this lesson's 'receiver down for 2 hours' challenge.

What

GitHub's checklist for webhook receivers.

Key idea

Use a secret to verify payloads, respond with 2xx within 10 seconds and process asynchronously, redeliver missed events and dedupe with the X-GitHub-Delivery header.

Why here

The receiver side of this lesson's design: fast acknowledgement, signature checks and idempotent processing.

What

The SRE book's description of Google's datacenter-wide cron service.

Key idea

Replicas elect a leader with Paxos and record each launch's state before and after starting it, so a new leader knows whether a job already ran.

Why here

This lesson's 'leader dies mid-job' scenario solved at Google: decide between skipping and double-running, and persist launch state to make that call.

What

How Slack replaced a single cron box with a scheduling service.

Key idea

One Kubernetes pod is elected leader to schedule jobs while others stand by; the work runs on Slack's Job Queue and a Vitess table dedupes and tracks each run.

Why here

The complete distributed cron design from this lesson: leader election for the scheduler, a queue for execution and a dedupe table for exactly-once runs.

What

PhonePe's central job scheduler, used for merchant settlements and coupon expiry.

Key idea

Clients submit one-off or recurring jobs that fire HTTP callbacks; jobs are stored durably and spread across workers, handling over 100K schedules per second.

Why here

Shows why this lesson centralises scheduling: embedded schedulers lose jobs on crashes and need per-service leader election, a platform solves it once.

What

How Shopify survived recurring flash sales, such as Kylie Cosmetics drops, that once took down a whole database shard.

Key idea

A checkout throttle written in Lua on its Nginx/OpenResty edge applies back pressure, admitting buyers at a rate checkout can handle.

Why here

The capstone's core move: protect inventory and checkout by queueing demand at the edge instead of letting it reach the database.

What

Why Shopify split its platform into pods after sharding its database in 2015.

Key idea

A pod is a set of shops on a fully isolated set of datastores, so a flash sale or failure in one pod cannot take down shops in another.

Why here

Applies failure-first thinking to the capstone: isolate the hot merchant's traffic so one sale cannot become a platform outage.

What

The original LinkedIn paper introducing Kafka.

Key idea

A topic is a partitioned, append-only log; consumers pull and track their own offsets, so messages stay readable after delivery and can be replayed.

Why here

The defining difference in this lesson: a queue deletes a message once handled, a stream keeps an ordered log that many consumer groups read independently.

What

How Slack rebuilt the Redis-based job queue that runs billions of background tasks.

Key idea

They put durable Kafka in front of Redis: a Go service (Kafkagate) enqueues to Kafka and a relay moves jobs into Redis for workers to execute.

Why here

A hybrid of both models from this lesson: a durable log absorbs bursts and outages, a work queue hands out tasks to workers.

What

Razorpay traces its UPI Switch from a monolith on SQS to a Kafka-based platform.

Key idea

SNS and SQS fan-out grew messy, Kafka fixed it, but one topic per event type plus retry and DLQ topics exploded partition counts until topics were consolidated.

Why here

Shows when a queue is enough and when a stream is worth it, and the operational cost (partitions per broker) this lesson's trade-off section warns about.

What

A complete walkthrough of the outbox pattern using Postgres, Debezium and Kafka.

Key idea

The service writes its business row and an outbox row in one transaction; Debezium reads the outbox from the database log and publishes to Kafka.

Why here

The exact pattern of this lesson, with log-tailing (CDC) as the relay instead of polling, and example code.

What

The pattern reference page for the transactional outbox.

Key idea

Updating a database and publishing a message cannot be atomic across two systems, so record the message in the same database and relay it afterwards.

Why here

A short statement of the dual-write problem this lesson opens with, plus the two relay options: polling publisher and transaction log tailing.

What

Netflix's change-data-capture framework for MySQL and Postgres.

Key idea

It streams committed changes from the transaction log and interleaves full-table dumps with log events using watermark rows, so dumps never overwrite newer changes.

Why here

Explains the CDC relay this lesson relies on and why reading the log beats dual writes from application code.

What

Netflix announces its adaptive concurrency limits library.

Key idea

Inspired by TCP congestion control, each service measures latency and adjusts how many requests it accepts at once, rejecting the excess quickly.

Why here

Backpressure without hand-tuned limits: the system detects saturation itself and pushes back, as this lesson's detection chapter describes.

What

Amazon's guide to keeping services healthy when offered more traffic than they can serve.

Key idea

Track goodput, not throughput: reject excess work early and cheaply, prioritise important requests and drop work whose caller has already timed out.

Why here

The 'shed load' response strategy from this lesson with the reasoning about where and how cheaply to reject.

What

Uber's library for shedding load by request criticality.

Key idea

A CoDel-inspired detector watches queueing delay per endpoint and, when a service degrades, drops lower-criticality requests first.

Why here

Adds priorities to backpressure: not all requests are equal, so overload control should protect the critical ones.

What

How Uber retries failed Kafka messages without blocking the main topic.

Key idea

Failed messages move to a chain of retry topics with growing delays and finally to a dead letter queue, so one bad message never stalls a partition.

Why here

The DLQ pattern of this lesson built on Kafka, where you cannot simply leave a poison pill at the head of the partition.

What

The SQS reference for dead-letter queues.

Key idea

A redrive policy moves a message to the DLQ after maxReceiveCount failed receives, and messages can later be redriven back to the source queue.

Why here

The managed-queue version of this lesson's DLQ: receive count as the poison-pill detector and redrive as the recovery path.

What

How Kafka Connect deals with records it cannot process.

Key idea

Choose between failing fast and tolerating errors; tolerated bad records go to a DLQ topic with headers describing the error for later inspection.

Why here

Shows the fail-fast vs skip-and-park decision this lesson asks you to make, and why error context must travel with the dead letter.

What

An explanation of the Merkle-tree logs behind Certificate Transparency and Go's checksum database.

Key idea

Hashing records into a Merkle tree lets anyone check that an entry is in the log and that the log was only appended to, never rewritten.

Why here

The tamper-evidence part of this lesson, beyond a simple hash chain: proofs that are small and quick to verify.

What

How AWS lets customers prove their CloudTrail audit logs were not altered.

Key idea

Each log file is hashed with SHA-256, and an hourly digest file listing those hashes is signed with RSA, so modification or deletion is detectable.

Why here

A production audit log using the hash-and-sign approach this lesson designs.

What

PhonePe's system for auditing human activity on its databases.

Key idea

Database audit logs stream into a pipeline that checks every query against rules, fingerprints events with SHA-256 to dedupe alerts, and keeps records for compliance.

Why here

Shows the consumer side of an audit log from this lesson: collecting, enriching and alerting on who did what, for regulators.

What

CRED's move from a monolithic order system to a central, domain-agnostic order platform.

Key idea

Separate business lines had built their own order systems, causing duplicate orders and inconsistent states; one platform OMS became the single source of order truth.

Why here

The capstone's order service at fintech scale: one owner of order state, with other services reacting to its events.

What

PhonePe's transaction store that powers payment status and history.

Key idea

The services that handle each step of a payment write to TStore, which consolidates them into one record per transaction (PENDING, then COMPLETED or ERRORED) for the app to read.

Why here

The capstone's tracking and audit needs: assemble one order timeline from many services' events instead of scattered reads.

What

Uber's retry-topic and DLQ design for Kafka consumers.

Key idea

Failures move to delayed retry topics and finally a DLQ, keeping the main order stream flowing while bad messages wait for a fix.

Why here

Ready-made answer to the capstone's failed-order deep dive: retry without blocking dinner-rush traffic, then park and alert.

What

An AWS whitepaper on putting Redis in front of a relational database.

Key idea

It compares lazy loading (cache-aside) with write-through, and covers TTLs, eviction policies and what to cache.

Why here

The four caching patterns of this lesson, with the same trade-offs: stale reads with cache-aside, wasted writes with write-through.

What

Amazon's lessons on when caches help and when they create new failure modes.

Key idea

Caches introduce modal behaviour: a service tuned to a warm cache can fall over when it is cold, so plan for cold starts, coalescing and negative caching.

Why here

Adds the operational view to this lesson's patterns: in-process vs external caches, and why the cache must not become a hidden dependency.

What

How Meta measured and improved cache consistency in TAO and Memcache.

Key idea

Invalidation races leave stale entries; a monitoring service (Polaris) checks cache against the database, which took TAO from six nines to ten nines of consistency.

Why here

Shows why this lesson calls invalidation the hard part: you must measure staleness, not assume invalidations always arrive in order.

What

Uber's Redis cache built into its Docstore database layer.

Key idea

TTL alone was too stale, so Flux tails the MySQL binlog (change data capture) and invalidates or refreshes cached rows shortly after each commit.

Why here

The hybrid strategy this lesson recommends: TTL as a safety net plus CDC-driven invalidation for freshness.

What

The paper describing how Facebook runs memcached as a look-aside cache for billions of requests per second.

Key idea

Leases let only one client refill a missing key while others wait or take slightly stale data, fixing both thundering herds and stale sets.

Why here

The lease technique from this lesson's protection strategies, in the system that popularised it.

What

A paper that analyses cache stampedes and proposes the XFetch algorithm.

Key idea

Each reader recomputes a value early with a probability that rises as expiry nears, so one request refreshes it before the herd arrives.

Why here

The formal basis for the probabilistic early refresh this lesson compares against locks and leases.

What

Fastly's documentation of how its CDN combines concurrent requests for the same object.

Key idea

Concurrent misses for one object become a single request to origin, and its response serves everyone waiting, at the cost of some queueing delay.

Why here

Request coalescing from this lesson applied at the CDN tier, including the cases where collapsing must be turned off.

What

Twitter's head of platform explains how home timelines were built.

Key idea

Tweets are fanned out on write into per-user timelines in Redis so reads are cheap; accounts with huge follower counts make that fan-out expensive.

Why here

The push model of this lesson in production, and the reason the hybrid approach exists.

What

LinkedIn's replacement for its Sensei-based feed infrastructure.

Key idea

FollowFeed builds a member's feed at request time by aggregating recent activity from the entities they follow, then ranks it, cutting p99 latency fivefold.

Why here

The pull model of this lesson at scale: indexes per author make fan-out on read fast enough for a dense network.

What

The paper on TAO, the read-optimised cache and store behind Facebook's social graph.

Key idea

Objects and associations are cached in leader and follower tiers in every region, serving a billion reads a second with eventual consistency.

Why here

The data layer under a feed: building a feed is mostly association lookups, and TAO caches exactly those.

What

A measurement study of every cache layer a Facebook photo passes through.

Key idea

Browser caches, edge caches and an origin cache each absorb part of the traffic before Haystack storage; smarter eviction such as S4LRU raises hit ratios.

Why here

This lesson's caching hierarchy measured end to end: what each tier catches and what is left for the next.

What

MDN's guide to how browsers and shared caches store HTTP responses.

Key idea

Cache-Control decides freshness, ETag and Last-Modified allow revalidation, and private vs shared caches decide who may store a response.

Why here

The browser and CDN tiers of this lesson are driven by these headers; getting them right is most of multi-tier invalidation.

What

Cloudflare's documentation of tiered caching between its data centres.

Key idea

Lower-tier data centres ask an upper tier before going to the origin, so most misses are served inside the CDN and origin load drops.

Why here

Shows that the CDN in this lesson's hierarchy is itself multi-tier, with its own hit-ratio trade-offs.

What

Amazon's lessons on running caches in front of services.

Key idea

A fleet-wide cache flush or a cold start can overload the backend; request coalescing and careful TTLs keep the service stable.

Why here

Directly useful for the capstone's thundering-herd deep dive on a global catalog after deploys or flash sales.

What

How Dream11 shows millions of users a personalised list of contests just before a match starts.

Key idea

Spark Streaming writes contests into Aerospike, and hot keys are seeded as several copies read by userId modulo copies, so no single node serves everyone.

Why here

Two capstone deep dives at once: personalised listings built from shared cached data, and hot-key spreading by key replication.

What

Amazon's CTO on what eventual consistency means for the clients of a replicated store.

Key idea

Client-side guarantees such as read-your-writes, session and monotonic reads can be offered on top of eventual consistency, and N, W and R quorums set the trade-off.

Why here

The consistency spectrum and quorum chapters of this lesson, explained by the person behind Dynamo.

What

Jepsen's clickable map of consistency models and how they relate.

Key idea

Models form a hierarchy, from strict serializable and linearizable down to causal, read-your-writes and eventual, each with what it forbids.

Why here

A reference for the consistency spectrum of this lesson, with each model's definition one click away.

What

A short paper that explains six consistency guarantees using the people who follow a baseball game.

Key idea

The scorekeeper, umpire, radio reporter and others each need a different guarantee (strong, bounded staleness, consistent prefix, monotonic reads, read-my-writes, eventual).

Why here

The best way into this lesson's main point: choose consistency per reader and per operation, not per database.

What

The 1987 paper that introduced sagas for long-lived transactions.

Key idea

Split a long transaction into steps that each commit locally, each paired with a compensating step that semantically undoes it if a later step fails.

Why here

The original definition behind this lesson: sagas trade isolation for availability and need compensations designed up front.

What

The pattern reference page for sagas in microservices.

Key idea

Choreography lets each service react to the previous one's events; orchestration has a coordinator tell services what to do, and both lack isolation.

Why here

A compact summary of this lesson's choreography vs orchestration chapter, with the isolation countermeasures it raises.

What

Netflix introduces Conductor, the orchestration engine for its content-processing workflows.

Key idea

Peer-to-peer choreography became hard to follow, so workflows are defined as JSON blueprints and a central engine schedules tasks on worker services.

Why here

Explains why a team moves from choreography to orchestration, the decision this lesson asks you to justify.

What

Uber's introduction to Cadence, with recorded talks from its creators Maxim Fateev and Samar Abbas.

Key idea

Workflow code runs as if it never fails: Cadence persists its state, queues and timers, so a process restart resumes the workflow where it was.

Why here

The ancestor of Temporal, the engine this lesson uses: durable execution as a replacement for hand-built saga state machines.

What

Temporal's documentation of what a Workflow is and how it executes.

Key idea

Workflows are deterministic code whose event history is recorded; on failure a worker replays the history to rebuild state, while side effects live in Activities.

Why here

The rules behind this lesson's Temporal code: why workflow code must be deterministic and why I/O goes in activities.

What

An annotated summary of a Systems @Scale 2021 talk on designing a workflow engine from first principles.

Key idea

Any workflow engine needs task queues, durable timers, consistency between state and queues, and sharding; Temporal's design follows from those needs.

Why here

The Temporal architecture chapter from the inside: history, matching and timers as answers to concrete requirements.

What

Werner Vogels' page publishing the full SOSP 2007 Dynamo paper.

Key idea

An always-writable store keeps concurrent versions, tracks them with vector clocks and hands siblings to the application to merge, like a shopping cart union.

Why here

The vector clock chapter of this lesson in its most famous production use, including why LWW was not enough.

What

SoundCloud's open-source store for the activity stream, built on Redis.

Key idea

A last-writer-wins element set CRDT keeps add and remove sets with timestamps, so replicas merge in any order and converge.

Why here

A small, readable production CRDT, and an example of LWW done per element rather than per object.

What

Figma's explanation of how simultaneous edits to one design file are merged.

Key idea

Figma rejected operational transforms; inspired by CRDTs, a central server orders changes and conflicts resolve last-writer-wins per object property.

Why here

A worked 'choose the right approach' decision from this lesson: when a central authority exists, a simpler CRDT-like scheme beats full OT.

What

A critique of the Redlock algorithm and a guide to safe distributed locks.

Key idea

A paused process can wake up still believing it holds a lock; only a fencing token checked by the storage layer stops its late writes.

Why here

The fencing token chapter of this lesson, with the GC-pause scenario that makes it necessary.

What

The Raft paper, written to make consensus easier to understand than Paxos.

Key idea

Consensus is split into leader election with randomised timeouts and terms, log replication from the leader, and safety rules about who can be elected.

Why here

The consensus chapter of this lesson; Raft is what etcd and many coordination services run under the hood.

What

Google's paper on Chubby, the lock service used by GFS and Bigtable to elect masters.

Key idea

A small Paxos-replicated cell offers coarse-grained locks and files with sessions and KeepAlives, plus sequencers that act as fencing tokens.

Why here

The coordination-service chapter in its original form: leader election as a library call on top of a lock service.

What

How Amazon uses leader election and where it goes wrong.

Key idea

Leases are the most widely used election mechanism at Amazon; they are built on measured durations rather than synchronised wall clocks, and leadership is usually sharded per item of data.

Why here

The capstone's matchmaker leader: shard queues so each has one leader, and make its actions safe if leadership flips.

What

The paper behind Xbox Live's skill rating and matchmaking.

Key idea

Each player's skill is a mean plus an uncertainty that shrinks with games played, updated by Bayesian inference after every match, including team games.

Why here

What the capstone's rating update actually computes, and why it must be applied exactly once per match.

What

AWS's documentation of its managed matchmaking service.

Key idea

Games send matchmaking requests to a matchmaker whose rule set defines team structure and player attributes such as skill and latency, then approves or rejects potential matches.

Why here

A production version of the capstone's matchmaking loop: requests in, rule-based evaluation, matches out to game servers.

What

The design of RENO, Netflix's system for pushing events to member devices.

Key idea

One event source, priority-specific queues and processing clusters, push to online devices plus pull for the rest, and stale events dropped by age.

Why here

A complete high-level design to compare with this lesson's: the same components (ingest, prioritise, route, deliver) with Netflix's reasoning for each.

What

How LinkedIn put one service in charge of every notification sent to members.

Key idea

ATC, built on Samza with state in RocksDB, decides the right message, channel, time and frequency for each member instead of every team sending on its own.

Why here

The reason this lesson's platform is centralised: without one decision point, teams flood users and nobody owns the total volume.

What

Razorpay redesigns its service for SMS, email and especially webhook notifications.

Key idea

API pods queue requests to SQS for workers, but the status database became the bottleneck; priority queues, rate limits per queue and DB changes fixed it.

Why here

The core flow of this lesson (send API, queue, workers, status store) under real load, and where it broke first.

What

How Uber replaced app polling with RAMEN, its push messaging platform.

Key idea

Services generate messages when state changes, and RAMEN delivers them to riders and drivers over long-lived streaming connections, later moved to gRPC.

Why here

The push channel worker of this lesson built in-house: connection management, delivery acknowledgement and fallbacks.

What

LinkedIn's streaming system for notifying members about content from their network.

Key idea

A batch Hadoop job that ran a few times a day was replaced by a near-real-time stream processor that scores and selects candidate notifications per member.

Why here

The batch vs real-time choice in this lesson's batching and digest chapters, and how relevance decides what is worth sending.

What

PhonePe's topic and subscription platform behind its chat, inbox and in-app instructions.

Key idea

Push notifications are often missed and not persistent, so messages are stored in topics that devices read, giving a durable in-app inbox.

Why here

The notification center of this lesson: a persistent inbox complements push so a dismissed alert is not lost.

What

How Netflix scaled Pushy, the WebSocket server that keeps connections open to devices.

Key idea

It handles hundreds of millions of concurrent connections and hundreds of thousands of messages a second at 99.999% delivery reliability.

Why here

The scale and reliability concerns of this lesson for the push channel: connection counts, reconnect storms and measuring delivery.

What

Hotstar's retrospective on traffic spikes during the 2017 Champions Trophy.

Key idea

A push notification sent as Virat Kohli walked in to bat took login traffic from 10K to 125K TPS in seconds, and the database had not been scaled for it.

Why here

The campaign-mode warning of this lesson: a mass notification is a load test on every system users land on.

What

The SRE book chapter on what to monitor and when to page a human.

Key idea

Watch the four golden signals (latency, traffic, errors, saturation) and page only on symptoms that need urgent, intelligent action.

Why here

The basis of this lesson's alerting chapter: alerts on user-visible symptoms such as delivery latency, not on every internal cause.

What

How Google's Borgmon turned time-series collection into the basis of monitoring and alerting.

Key idea

Export counters from every service, aggregate them with rules and alert on rates and ratios computed from the time series.

Why here

The metrics pillar of this lesson in depth; Prometheus follows the same model.

What

How Zerodha monitors a mix of bare-metal, AWS and Kubernetes systems for a large share of India's retail trading.

Key idea

Prometheus labels and PromQL let one stack filter and aggregate metrics across dissimilar environments, feeding dashboards and alerts.

Why here

A compact, real setup for the dashboards and alerts this lesson builds for a notification platform.

What

Swiggy breaks up the 'delivery monolith' behind its three-sided food marketplace.

Key idea

The old model treated every delivery as one pick-up and one drop; the new model of tasks, jobs, job legs and trips supports multi-leg deliveries across Food, Stores and Genie.

Why here

The capstone's order and driver-assignment flows: model delivery as a state machine of tasks so new business lines reuse it.

What

How Uber rebuilt the platform that tracks every trip and courier session.

Key idea

The old design favoured availability, with Ringpop serialising writes per key; the new one moved trip state to a NewSQL store with strongly consistent transactions across regions.

Why here

The capstone's distributed-coordination deep dive: who owns a driver's state, and why consistency mattered more as products multiplied.

What

A section of the standard information retrieval textbook, free online.

Key idea

Tokenise and normalise documents, sort term and document ID pairs, then group them into a dictionary of terms pointing to sorted postings lists.

Why here

The inverted index of this lesson built step by step, and why it beats SQL LIKE scans.

What

A tour of Elasticsearch internals starting from the inverted index and working upwards.

Key idea

An index is made of Lucene shards, each a set of immutable segments that are searched together and merged in the background.

Why here

Explains the 'how Elasticsearch works' chapter: segments, refresh and merges are why indexing is near-real-time, not instant.

What

How Yelp made tens of millions of orders searchable without loading MySQL.

Key idea

Its real-time data pipeline streams changes from MySQL into Elasticsearch, so search reads a denormalised copy while MySQL stays the source of truth.

Why here

The CDC-driven indexing pipeline of this lesson, including keeping the index in sync and backfilling.

What

The classic recipe for changing an Elasticsearch mapping on a live index.

Key idea

Point the application at an alias, build a new index with the new mapping, reindex into it, then swap the alias atomically.

Why here

The reindexing problem of this lesson; the post's editor note points to the newer Reindex API for the same flow.

What

How GitHub built Blackbird, a search engine in Rust for all public code.

Key idea

Kafka events trigger crawlers, content is sharded by Git blob ID so duplicate files are indexed once, and ngram indexes support substring search.

Why here

An indexing pipeline far beyond Elasticsearch defaults: dedupe before you index and choose the shard key for even load.

What

A term-by-term walk through the BM25 scoring formula.

Key idea

IDF rewards rare terms, k1 makes term frequency saturate and b normalises for field length; tuning them changes ranking.

Why here

The BM25 chapter of this lesson in more depth, with what each parameter does to real results.

What

Airbnb's account of moving search ranking from gradient boosted trees to neural networks.

Key idea

Gains from the tree model plateaued; the paper describes the journey to deep models, including what failed and the features that mattered.

Why here

Where relevance goes after this lesson's BM25 and boosts: a second-stage ranker trained on bookings.

What

How Zomato unified search across restaurants, dishes, cuisines and collections.

Key idea

Search guesses both the entity and the intent behind a query, mixing explicit signals (what is typed) with implicit ones such as location and time.

Why here

Relevance when one box searches restaurants, dishes and cuisines: this lesson's multi-field search plus intent signals.

What

A short essay that builds a working spelling corrector in about 20 lines of Python.

Key idea

Generate candidates within one or two edits, then pick the one with the highest probability under a language model built from a text corpus.

Why here

The 'did you mean?' chapter of this lesson from first principles: edit distance for candidates, frequency for ranking.

What

LinkedIn open-sources the engine behind its typeahead search.

Key idea

Prefix inverted lists find candidates, per-document Bloom filters quickly reject non-matches, and a forward index removes false positives before scoring.

Why here

A low-latency autocomplete design to set beside this lesson's trie and edge n-gram options.

What

A Flipkart talk on ranking autocomplete suggestions for short prefixes such as 'red'.

Key idea

Recent queries in the session reveal intent, so suggestions are re-ranked per user, over 5 million documents at 10K QPS within 20 ms.

Why here

Combines two chapters of this lesson, autocomplete and intent, for Indian e-commerce at sale-day scale.

What

Elastic's official guidance on how many shards to create and how big they should be.

Key idea

Too many shards (oversharding) waste resources and destabilise the cluster, while very large shards slow searches and recovery; plan a sharding strategy.

Why here

The cluster-sizing arithmetic of this lesson, from the vendor's own production guidance.

What

How PhonePe's analytics platform indexes 100 billion events a day into Elasticsearch.

Key idea

Daily indices, a custom job that spreads today's hot shards evenly across nodes, and shards of about 15-20 GB keep millions of writes per second balanced.

Why here

Real operational fixes for the skew and shard-size problems this lesson's scale chapter raises.

What

Yelp's migration of its core business search from a custom Lucene system to Elasticsearch.

Key idea

The legacy engine was sharded distributed Lucene with master-slave replicas; every code push meant cycling workers that had to re-download and re-warm their Lucene indexes, so pushes were slow.

Why here

The operations side of this lesson: why teams trade a hand-built search stack for a managed topology.

What

GitHub's Blackbird search engine for code.

Key idea

Code needs exact punctuation, no stemming and regular expressions, so GitHub built ngram indexes and a query engine that intersects posting iterators.

Why here

Like legal text, code breaks default analysers; the capstone's legal query parser faces the same 'exact phrase and citation' needs.

What

How Dream11's DevOps team runs Elasticsearch as a shared service for many teams.

Key idea

One platform serves logging, in-app search, WAF security logs and anomaly detection at over 500,000 events per second.

Why here

The capstone's multi-tenant deep dive: many tenants on shared clusters need sizing, isolation and per-tenant indices.

What

How PhonePe moves about 100 billion events a day into Kafka.

Key idea

A client library writes events to local disk (BigQueue) and a lightweight Ingestor process ships them to an ingestion service, which writes to Kafka.

Why here

The ingestion tier of this lesson with a local buffer: apps keep logging even when Kafka or the network is slow.

What

Segment's deduplication system for the events customers send to its API.

Key idea

About 0.6% of events arrive twice; workers check each message ID against embedded RocksDB within a de-duplication window before re-publishing to Kafka.

Why here

The validation and dedupe step of this lesson's pipeline: clients retry, so ingestion must drop duplicates by event ID.

What

How LinkedIn operated Kafka in 2015.

Key idea

Over 13 million messages a second ran through more than 1,100 brokers in 60+ clusters, with tiered clusters and mirroring between datacentres.

Why here

The Kafka backbone of this lesson at real scale, including how clusters are split and aggregated.

What

The essay that proposed the Kappa architecture as an alternative to Lambda.

Key idea

Keeping batch and streaming code in sync is the real cost of Lambda; instead keep the log and reprocess history with the stream processor when logic changes.

Why here

The Lambda vs Kappa choice of this lesson, argued by Kafka's co-creator.

What

A Google engineer's introduction to stream processing terms and patterns.

Key idea

Streaming is processing unbounded data, and done right it can be as correct as batch; event time and processing time must be kept apart.

Why here

Reframes this lesson's 'two worlds' as one: batch is a special case of streaming over bounded data.

What

The paper that introduced MapReduce.

Key idea

Express a job as map and reduce functions and the framework handles partitioning, scheduling, shuffling and re-running failed tasks across thousands of machines.

Why here

The ancestor of the Spark batch layer in this lesson, and still the clearest account of why batch is simple to reason about.

What

The Kimball Group's index of dimensional modelling techniques.

Key idea

Fact tables at a declared grain, dimension tables around them, and named patterns for slowly changing dimensions (Type 1, 2, 3 and others).

Why here

The source of this lesson's star schema and SCD vocabulary, with a short page per technique.

What

The paper behind Google's interactive query engine, later BigQuery.

Key idea

A columnar layout for nested records plus multi-level execution trees runs aggregations over trillion-row tables in seconds.

Why here

Why this lesson's columnar storage chapter matters: analytic queries read few columns, so store and scan by column.

What

The sequel to Streaming 101, with animated examples of windowing over out-of-order data.

Key idea

Watermarks estimate event-time completeness, triggers decide when to emit results, and allowed lateness decides how long to keep refining them.

Why here

The watermark, windowing and late-data chapters of this lesson, explained with the same 'what, where, when, how' questions.

What

The paper that formalised Google Cloud Dataflow's model, the basis of Apache Beam.

Key idea

Windowing, triggering and accumulation modes are separate knobs, letting a pipeline trade correctness, latency and cost for late, out-of-order data.

Why here

The theory under this lesson's reprocessing and correction choices: discarding, accumulating or retracting late updates.

What

How Flipkart Ads counts impressions and clicks against advertiser budgets in real time.

Key idea

A stateful Flink pipeline uses event-time semantics, deduplication and watermarking, because lag means overspending and blocking means lost revenue.

Why here

Late and duplicate events with money attached: this lesson's correctness trade-offs where a wrong count is a refund.

What

Cloudflare replaces its Postgres-based analytics pipeline with ClickHouse.

Key idea

The old pipeline rolled aggregates up by minute, hour and day in Postgres; the new one feeds a raw requests table and ClickHouse materialized views hold minutely aggregates per breakdown.

Why here

The OLAP choices of this lesson in one migration: pre-aggregation, columnar engines and schema design for the queries you serve.

What

PhonePe's real-time event analytics platform built on Apache Druid.

Key idea

Druid rolls data up at ingestion (about 5:1), uses sketches for distinct counts and percentiles, and runs active-active across two data centres.

Why here

Pre-aggregation and approximate queries from this lesson, tuned for sub-second dashboards.

What

Netflix's platform for producing, routing and processing event streams.

Key idea

Keystone offers a Kafka-based data pipeline that routes events to sinks per user config plus self-serve stream processing, handling trillions of events a day.

Why here

The capstone's ingestion and processing layers at the scale of the service the capstone imitates.

What

How Dream11 tracks contest joins, payment gateway health and campaigns in real time.

Key idea

Debezium and Kafka Connect pull data from transactional databases into Kafka, KSQL enriches and joins streams, and sinks feed real-time stores and alerts.

Why here

The capstone's real-time trending and alerting path, built from CDC plus stream SQL.

What

AWS's reference definitions of the three SaaS isolation models.

Key idea

Silo gives each tenant dedicated resources, pool shares everything with tenant-scoped access, and bridge mixes the two per layer of the stack.

Why here

The isolation patterns of this lesson under the names most SaaS architects use: database-per-tenant, shared schema and hybrids.

What

A step-by-step guide to isolating tenants in one shared Postgres schema.

Key idea

RLS policies compare each row's tenant_id with a session setting, so the database itself filters every query even if application code forgets to.

Why here

The row-level security implementation of this lesson, with the session-context plumbing it needs.

What

How Amazon keeps shared services fair when one tenant's traffic spikes.

Key idea

Rate limiting (admission control), per-tenant quotas and workload isolation stop one customer's load from degrading others, with care about where limits are enforced.

Why here

The quota design principles and rate limiting chapters of this lesson, from services that serve millions of tenants.

What

The technique Route 53 uses to contain the damage from one abusive customer.

Key idea

Give each customer a near-unique combination of a few workers out of many, so a customer that overwhelms its shard rarely shares all of them with anyone else.

Why here

A stronger form of the resource isolation in this lesson: limit blast radius without one silo per tenant.

What

Cloudflare's product for customers who must keep traffic, keys and logs in a region.

Key idea

Regional Services decides where traffic is decrypted and inspected (elsewhere it only gets L3/L4 DDoS protection and is sent on securely), and Geo Key Manager decides where private keys live.

Why here

Data residency for a global edge network: decide separately where data is processed, where keys live and where logs are stored.

What

The April 2018 RBI circular on where payment data may be stored.

Key idea

All system providers must store the entire data relating to payment systems they operate, with end-to-end transaction details, only in India.

Why here

A non-GDPR residency rule that shaped every Indian payments architecture, and an example of how a regulation becomes a design constraint.

What

Facebook's deletion framework, described in a USENIX Security paper.

Key idea

Developers annotate how data types relate, DELF cascades deletions along those edges across stores, and checks catch data that would be orphaned or wrongly deleted.

Why here

The data inventory and cascading deletion chapters of this lesson at social-network scale.

What

Google's description of what happens when customer data is deleted.

Key idea

Deletion moves through stages: a request, soft deletion with a recovery period (up to 30 days in some cases), logical deletion from active systems, then expiry from backups; cryptographic erasure also applies.

Why here

The staged deletion service of this lesson, including the backup problem: you delete from backups by letting them expire.

What

Google's paper on removing the privileged corporate network.

Key idea

Access is decided per request from device state and user identity, not from being inside the firewall, so the network perimeter stops being a trust boundary.

Why here

Zero-trust thinking for this lesson's trust boundary chapter: authenticate and authorise every hop.

What

Google Cloud KMS's explanation of envelope encryption.

Key idea

Data is encrypted with a data encryption key (DEK), and the DEK is encrypted with a key encryption key (KEK) held in KMS; use a fresh DEK per write and never share one across users.

Why here

The key hierarchy of this lesson's encryption strategy, and why per-tenant keys make crypto-shredding possible.

What

Atlassian's review of the outage in which 775 customers lost access to their products for up to 14 days.

Key idea

A deletion script accepted IDs without confirming whether they were apps or whole sites, so sites were deleted; restoring them from shared multi-tenant storage took up to 14 days.

Why here

The capstone's isolation and deletion deep dives in one incident: tenant-scoped restore must be designed in, not improvised.

What

How Slack moved Enterprise Grid customers from a workspace-centric to an organisation-centric architecture.

Key idea

Users in many workspaces had to switch views and saw sync bugs, so data access was re-oriented around the user and organisation instead of one workspace.

Why here

The capstone's tenant model: decide early whether the tenant is the org or the sub-unit, because changing it later is a re-architecture.

What

The SRE book chapter that defined SLIs, SLOs and SLAs for most of the industry.

Key idea

Pick indicators users actually feel, set targets below 100%, and keep SLAs (contracts with penalties) looser than internal SLOs.

Why here

The vocabulary of this lesson at its source, including how to choose good SLIs.

What

A practical guide to rolling out SLOs, with a worked example service.

Key idea

Measure SLIs where users experience them, agree an error budget policy in advance, and use the budget to decide between reliability work and features.

Why here

The error budget and SLO-based decision-making chapters of this lesson, step by step.

What

The workbook chapter on turning SLOs into pages.

Key idea

Alert on how fast the error budget is burning, over multiple windows, so fast outages page quickly and slow leaks still get caught without noise.

Why here

The step after setting SLOs in this lesson: burn-rate alerting is how an SLO reaches an on-call engineer.

What

The Google paper that introduced trace trees and spans.

Key idea

Propagate a trace ID through every RPC, record spans with low overhead and sample aggressively so tracing can stay on everywhere in production.

Why here

The design behind the tracing chapter of this lesson; Zipkin, Jaeger and OpenTelemetry follow its model.

What

The history of tracing at Uber that led to Jaeger.

Key idea

Uber went from Merckx, its first tracer built on a Kafka stream, through Zipkin-style instrumentation to Jaeger, and switched from a pull to a push architecture.

Why here

What it takes to roll tracing out across hundreds of services, the hard part this lesson's observability stack glosses over.

What

Why Zerodha moved its application logs off the ELK stack.

Key idea

Elasticsearch struggled with post-2020 volumes and long regulatory retention, so logs now flow through Vector into ClickHouse, with Metabase for queries.

Why here

The logs pillar of this lesson with real constraints: volume, cost and retention decide the storage engine.

What

Netflix's open-source system for judging canary releases automatically.

Key idea

Spinnaker deploys a new baseline cluster and a canary side by side, and Kayenta compares their metrics statistically to pass or fail the release.

Why here

The canary chapter of this lesson made rigorous: compare against a fresh baseline, not the old fleet, and let statistics decide.

What

Dream11's automated blue-green deployment platform.

Key idea

OneClick creates the green stack sized from a predicted-concurrency model, then shifts traffic to it gradually instead of all at once.

Why here

Blue-green from this lesson with a twist: size the new stack for upcoming traffic and move traffic in steps.

What

How Stripe changes data models under a live payments API.

Key idea

Four steps: dual-write to old and new, change reads to the new path, change writes to the new path, then remove the old data.

Why here

The expand-and-contract database migration of this lesson, done with no downtime on critical data.

What

Facebook's system for load testing with real user traffic.

Key idea

Shift live traffic onto a cluster or region until it reaches a target load, watch health metrics, and record the bottlenecks found along the way.

Why here

The load-testing chapter of this lesson without synthetic traffic: measure real capacity by steering production load.

What

How Dream11 scales for traffic that spikes around each match's toss and start.

Key idea

With 5.5 million concurrent users and 100 million requests a minute at peak, scaling follows the match timeline (before, during, after) rather than waiting for autoscaling.

Why here

Forecast-driven scaling from this lesson: when you know when the spike comes, provision ahead of it.

What

Hotstar's account of a capacity miss during an India-Pakistan match.

Key idea

The fleet was scaled for 7M concurrent users, but a sudden login spike hit a database that had not been scaled in proportion, since logins cannot be cached.

Why here

The bottleneck lesson in one story: capacity plans must scale every tier on the hot path, not just the stateless front.

What

The SRE book chapter on running an incident.

Key idea

Separate roles (incident commander, operations, communications, planning), keep a live incident document and hand off command explicitly.

Why here

The incident response fundamentals of this lesson, with an example of an incident managed badly and then well.

What

Atlassian's public handbook for how its teams detect, respond to and learn from incidents.

Key idea

It defines what counts as an incident, sets values for each stage (know before customers do, escalate early), describes the incident manager role and ends in postmortems that assign actions.

Why here

A complete process to borrow for this lesson's on-call, runbook and postmortem sections.

What

Cloudflare's detailed postmortem of a global outage.

Key idea

A new WAF rule contained a regular expression that backtracked badly and exhausted CPUs on every HTTP-serving core worldwide; the fix list covers testing, rollout and regex engines.

Why here

A model postmortem for this lesson: timeline, root cause, why safeguards failed and concrete follow-ups.

What

What happens behind one ad break on JioHotstar, from ad formats to the decision engine.

Key idea

An ad decision engine picks an ad in milliseconds, mixing direct deals with programmatic auctions through SSPs, and inserts it server-side or client-side.

Why here

The capstone's ad-serving request path and latency budget, told by an Indian streaming platform.

What

How Flipkart Ads deducts impressions and clicks from advertiser budgets as they happen.

Key idea

Spend is aggregated with event-time semantics, dedupe and watermarks; lagging means overspend, blocking means lost revenue.

Why here

Exactly the capstone's budget-management deep dive: stop serving before the budget runs out without double-charging.

What

Google's system for joining ad queries with clicks in real time across datacentres.

Key idea

Photon joins unordered, delayed streams with no duplicates in the output at any time and exactly-once results eventually, surviving whole-datacentre outages.

Why here

The capstone's billing and audit path: clicks must be joined to impressions exactly once, because each one is money.

What

The story of the messaging layer inside Razorpay's UPI Switch, which talks to NPCI for most of its UPI volume.

Key idea

UPI is asynchronous, so a message bus sits at the heart of the switch; SQS fan-out gave way to Kafka, then topics were consolidated when partitions exploded.

Why here

What 'bank integration at scale' looks like inside a payment service provider: every UPI callback is an event fanned out to NPCI updates, merchant callbacks and the warehouse.

What

PhonePe's transaction store behind the green success screen and the payment history feed.

Key idea

Every service in a payment flow writes to TStore, which keeps one consolidated record per transaction and ingests around 45,000 writes a second at peak.

Why here

The UPI app side of this quest: a pending payment must show one consistent status even while the bank and NPCI are still answering.

What

A short 2012 WhatsApp post showing one server holding over 2 million TCP connections.

Key idea

A single FreeBSD box running Erlang held 2.27 million open sockets with CPU to spare, the efficiency behind WhatsApp's tiny server fleet.

Why here

The 'Why Erlang?' deep dive of this quest in one screenshot: lightweight processes per connection and a tuned kernel.

Our link checker received HTTP 400 (bot filtering); the page was opened and read with a second fetcher on 2026-10-18.

What

How WhatsApp let up to four companion devices work without the phone, keeping end-to-end encryption.

Key idea

Each device has its own identity keys, and senders encrypt every message separately for each of the recipient's devices (client fan-out).

Why here

The Signal-protocol deep dive of this quest extended to many devices per user, and why the server never sees plaintext.

What

WhatsApp's rollout of automatic checks that the public keys it hands out are genuine.

Key idea

Public keys are kept in an Auditable Key Directory whose changes can be verified with proofs, so a swapped key would be detectable.

Why here

Closes the gap in this quest's encryption design: end-to-end encryption is only as strong as the key directory.

What

UIDAI's own 2014 description of the Aadhaar system, kept on the Internet Archive.

Key idea

Openness, linear scalability, security and vendor neutrality; three ABIS vendors deduplicate every enrolment, on open-source stacks with Hadoop, HBase and MongoDB.

Why here

The primary source for this quest's multi-ABIS deduplication and enrolment pipeline.

UIDAI no longer hosts this PDF; the Internet Archive copy was opened and its text checked on 2026-10-18.

What

UIDAI's specification for apps and portals that authenticate residents through the Aadhaar app.

Key idea

An authenticating agency's app or portal calls the Aadhaar app by app-to-app intent or QR code for face authentication and credential exchange, on Android and iOS.

Why here

The current face-authentication path from this quest's deep dive, as UIDAI documents it today.

What

Stripe's immutable, auditable log of every money movement.

Key idea

Ledger records double-entry style balance changes for every payment step and continuously checks that funds flows close out, flagging anything unexplained.

Why here

The clearing and settlement side of this quest: authorisation is fast, but the money trail must reconcile exactly afterwards.

What

Visa's own overview of VisaNet, the network that authorises card transactions.

Key idea

Banks and partners connect to VisaNet to authorise transactions, merchants reach it through them, and Visa's 'six nines' target allows under a second of downtime a day.

Why here

The four-party model and availability target this quest designs for, in the network operator's words.

What

How PhonePe's Trust & Safety team built a model that declines fraudulent peer-to-peer payments as they happen.

Key idea

Fraud reports are noisy and mixed with disputes, so they segmented reported cases with demographic, behavioural and transaction signals before training a real-time model.

Why here

The fraud engine in this quest's authorisation path: the decision must be made in-line, from features available in milliseconds.

What

The paper describing the original Google prototype.

Key idea

Crawling, a forward and inverted index split into barrels, and PageRank computed from the link graph to rank results.

Why here

The crawl, index and rank pipeline of this quest at its origin, including the PageRank deep dive.

What

The paper on Percolator, the system that let Google update its index continuously.

Key idea

Cross-row transactions and notifications on top of Bigtable let each crawled document update the index incrementally instead of rebuilding it in batch.

Why here

The indexing pipeline of this quest when freshness matters: incremental updates instead of periodic full rebuilds.

What

Google's public explanation of how Search ranks results.

Key idea

Ranking weighs many signals (meaning of the query, relevance, quality, usability, context) and the weight of each depends on the query, such as freshness for news.

Why here

The serving layer of this quest: BM25-style relevance is one input among many, and query intent decides how they are mixed.

What

Netflix's technical overview of Open Connect, the CDN it began building in 2011.

Key idea

Open Connect Appliances sit inside ISP networks and at exchange points, are filled with predicted-popular titles in off-peak windows, and AWS control-plane services steer each client to the best appliance.

Why here

The data plane of this quest's two-plane architecture: video comes from appliances next to the viewer, while AWS only decides which one.

What

How Netflix stopped using one bitrate ladder for every title.

Key idea

Encoding each title at many settings and measuring quality against bitrate gives a custom ladder: simple animation needs far fewer bits than grainy action.

Why here

The per-title encoding section of this quest's ABR deep dive, with the quality curves behind it.

What

How Netflix made its control plane survive the loss of a whole AWS region.

Key idea

Both regions serve traffic, DNS (Denominator) and Zuul shift users between them, and Cassandra replicates data across regions.

Why here

The control plane half of this quest's architecture, and why the microservices behind 'play' must run in more than one region.

What

The first post in Spotify's series on rebuilding the pipeline that collects client events.

Key idea

Every play and search becomes an event; the old Kafka and Hadoop pipeline is explained, and why Spotify chose Google Cloud managed services for the new one.

Why here

The event processing deep dive of this quest: play events feed royalties and recommendations, so delivery must be complete.

What

How Spotify's personalisation stack served recommendations for 60 million users.

Key idea

Kafka collects logs, Storm processes them in real time and Crunch runs batch jobs on Hadoop, and Cassandra serves the resulting user profiles at low latency.

Why here

The hybrid recommendation architecture of this quest: batch and streaming models meeting in one serving store.

What

Spotify explains Backstage, its developer portal, the day after open-sourcing it.

Key idea

A single portal catalogues every service, its owners and docs, with plugins for tooling, so thousands of engineers can find and run things in one place.

Why here

The 'Microservices & Backstage' deep dive of this quest, from the team that built it.

What

A guest post by S3's distinguished engineer, based on his FAST 2023 keynote.

Key idea

Drives keep growing while their I/O stays flat, so S3 manages 'heat' (requests per disk) by spreading each customer's data over very many drives to avoid hotspots.

Why here

The storage layer of this quest seen from inside: why placement and spreading data over many drives matters for durability and speed.

What

How S3 became strongly consistent for every request in 2020 without a performance hit.

Key idea

S3 kept its metadata cache but added a cache coherence protocol: a new witness component is notified of every write and tells readers when the cached view is stale.

Why here

The strong consistency deep dive of this quest, with the real mechanism S3 used.

What

AWS's postmortem of the February 2017 S3 outage in US-EAST-1.

Key idea

A playbook command removed more servers than intended, taking out the index subsystem that holds all object metadata, which then needed a slow full restart.

Why here

Why this quest partitions the index into cells: one oversized subsystem took a whole region of S3 down.

What

Uber introduces H3, its open-source hexagonal grid system.

Key idea

Hexagons have uniform neighbour distances and nest across resolutions, which suits analysing supply, demand and pricing by area.

Why here

The H3 deep dive of this quest at its source, including why hexagons beat squares for surge pricing cells.

What

How Uber improved its ETA predictions with a deep learning model.

Key idea

A routing engine computes an ETA from map data and live traffic, and a deep model predicts the residual correction on top of it.

Why here

The ETA deep dive of this quest: physical model plus learned correction, served at very high request rates.

What

Uber's rebuild of the platform that holds the state of every trip and driver session.

Key idea

The old Ringpop-based design favoured availability; the new one moved to a NewSQL store with strongly consistent transactions to support many products.

Why here

The matching flow of this quest needs one source of truth for each trip and driver, which is what this platform provides.

What

Twitter's head of platform on how home timelines were served.

Key idea

Tweets fan out on write into Redis-held timelines so reads are cheap, while accounts with enormous follower counts make write fan-out costly.

Why here

The fan-out and celebrity-merge deep dives of this quest, from the system it is modelled on.

What

Twitter's 2023 explanation of how the For You timeline is built.

Key idea

Candidate sourcing pulls the best tweets from several sources, a machine-learning model ranks them, and heuristics and filters finish the feed in a service called Home Mixer.

Why here

The timeline service of this quest after fan-out: ranking turns a list of tweet IDs into a feed.

blog.x.com answers automated checkers with HTTP 403; the page was verified through an Internet Archive snapshot taken 2026-03-10.

What

The open-sourced code of X's recommendation system.

Key idea

Separate services and jobs, such as SimClusters community embeddings and the timeline ranker, power For You, Search, Explore and Notifications.

Why here

Lets you see the social-graph and ranking services from this quest as real modules.

What

How Cloudflare built its Waiting Room product, first offered for COVID-19 vaccination sites.

Key idea

When users exceed a site's limit, new arrivals get a cookie and a queue page with an estimated wait; Workers and Durable Objects keep global counts to admit them in order.

Why here

The virtual waiting room deep dive of this quest: hold the herd at the edge and admit at the rate the booking system can take.

What

How Shopify kept recurring flash sales from taking down shared database shards.

Key idea

A checkout throttle in Lua on the Nginx edge applies back pressure, letting buyers into checkout only as fast as it can process them.

Why here

Tatkal at 10:00 is a flash sale: the same edge throttle protects the seat inventory rows this quest calls the hottest in the database.

What

How Instacart keeps item availability consistent across search, item pages and checkout.

Key idea

ML availability scores are stored in the database and refreshed two ways: a full sync several times a day and a lazy refresh triggered when items appear in search.

Why here

The real-time inventory deep dive of this quest: availability shown to a customer must agree on every surface.

What

How Zepto turns noisy rider GPS pings into the paths riders actually took.

Key idea

Outlier pings are removed, stalls detected, gaps smoothed with a Kalman filter, and the result map-matched with OSRM and given a confidence score.

Why here

The live tracking and promise-engine inputs of this quest: ETAs and SLAs are only as good as the location data under them.

What

Uber's open-source hexagonal grid for indexing locations.

Key idea

Every point maps to a hexagon at a chosen resolution, with uniform neighbour distances and parent-child nesting.

Why here

A practical way to precompute which dark store serves each delivery cell, the serviceability lookup in this quest.

What

Zerodha's CTO on the principles behind running India's largest stock broker with a small team.

Key idea

Keep it simple, find the global bottleneck (usually the database) before micro-optimising, and trust a well-partitioned RDBMS before reaching for exotic tools.

Why here

The mindset behind the Kite design in this quest, from the people who run it.

What

Martin Fowler's description of LMAX, a retail trading platform processing millions of orders a second on one thread.

Key idea

Business logic runs single-threaded in memory, fed by Disruptor ring buffers that handle input, journaling and replication around it.

Why here

The order path of this quest pushed to the extreme: sequence orders, keep state in memory and rebuild it from the journal.

What

How Zerodha generates, signs and emails over 1.5 million contract notes after market close.

Key idea

A chain of jobs (generate, sign, email) runs across workers through Tasqueue, Zerodha's lightweight Go job library, using EC2 and S3 for scratch storage.

Why here

The after-the-bell half of a trading day in this quest: a regulatory deadline turns into a distributed batch job.

What

The vLLM paper from UC Berkeley, which introduced PagedAttention.

Key idea

The KV cache is stored in fixed-size blocks mapped like virtual memory pages, which cuts fragmentation and lets requests share prefixes, raising throughput.

Why here

The paged KV memory and prefix caching deep dive of this quest at its source.

What

The paper that introduced iteration-level scheduling for LLM serving.

Key idea

Instead of waiting for a whole batch to finish, the scheduler adds and removes requests at every decoding step, so short answers do not wait for long ones.

Why here

Where this quest's batching deep dive comes from: continuous batching is iteration-level scheduling.

What

A benchmark-driven explainer of LLM inference batching.

Key idea

LLM serving is memory-bound, so batching is the main lever; continuous batching and paged memory raised throughput up to 23x in their tests while lowering median latency.

Why here

The back-of-the-envelope GPU maths of this quest checked against real measurements.

What

Figma's co-founder explains how simultaneous edits to a file are merged.

Key idea

Rather than operational transforms, a server per document orders changes and resolves conflicts last-writer-wins per property of each object, an approach inspired by CRDTs.

Why here

The data model and conflict-rules deep dive of this quest, from the system it is modelled on.

What

How Figma keeps the order of layers consistent when several people reorder them at once.

Key idea

Each child stores a fractional index between its neighbours, so moving a layer changes one value and concurrent moves never conflict on a shared array.

Why here

The 'parent pointers and fractional indexing' part of this quest's data model.

What

How Figma stopped losing up to a minute of edits when a multiplayer server crashed.

Key idea

Files used to be checkpointed every 30-60 seconds; a write-ahead log (the journal) now saves edits as they arrive, so recovery replays the journal on top of the last checkpoint.

Why here

The persistence and one-owner-per-file deep dive of this quest: in-memory state needs a log behind it.