Day 1Consistency Models

Who sees a write, and when? A write lands on Node 1 and a read hits Node 2 before replication arrives. Consistency models are precise answers to what that read may return: new value, old value, or an error.

The spectrum, strongest first. Linearizable (behaves like one copy), sequential (one agreed order), causal (cause before effect), read-your-writes, monotonic reads (never go backwards), eventual (converges, no promise when).

Session guarantees users notice. Read-your-writes: route a user's reads to the primary after their write, or track the write version. Monotonic reads: remember the last version read and only accept replicas at or past it.

Choose per operation. Browse products eventually consistent, reserve inventory and check out strongly consistent, order history read-your-writes. DynamoDB and Cassandra let you choose per read: eventual, QUORUM, ALL.

Spanner buys strong consistency with time. TrueTime (GPS and atomic clocks, uncertainty typically under 7 ms) plus Paxos and commit-wait give global linearizability, at ~10–20 ms per write and 100 ms+ across regions.

• Reading from a replica right after writing to the primary: the user's change "disappears".
• Random replica per read, so a counter appears to go backwards.
• Strong consistency for everything: latency and availability pay for reads that don't need it.

Day 2Distributed Transactions — The Saga Pattern

Why not two-phase commit. 2PC holds locks from PREPARE until the decision, blocks if the coordinator dies, needs every service up at once, and most NoSQL stores, brokers like Kafka and third-party APIs (Stripe) can't take part.

A saga is local steps plus compensations. Each step is a local transaction in one service; if step N fails, run compensations for steps 1 to N-1. No distributed locks, no coordinator holding them.

Compensation is not rollback. A refund is a new transaction that reverses the charge; both stay in history. Compensations must be idempotent, check state first (no-op if the step never ran) and eventually succeed.

The pivot transaction. The point of no return, like shipping. Before it, recover backward (compensate); after it, recover forward (retry until done). In the Airbnb example the pivot is capturing payment.

Choreography vs orchestration. Choreography: services react to each other's events; fine for 2–4 steps but the flow is hard to see. Orchestration: a coordinator calls each step and persists saga state; better for 5+ steps and branching.

• A refund compensation that can fail with no retry: the saga sticks with the customer charged.
• Payment steps without an idempotency key like saga-{id}-payment: a timeout retry charges twice.
• Compensating everything blindly instead of tracking which compensations finished.

Day 3Saga Orchestration Deep Dive

Durable execution. Temporal records every activity result in an event history. After a crash the workflow replays from the start, completed activities return their stored results instead of running again, and execution continues where it stopped.

Workflows decide, activities act. Workflow code must be deterministic because it replays: no random numbers, clock reads, UUIDs or HTTP calls (use workflow.now(), workflow.random(), activities). Activities do the side effects, with timeouts and retries.

Signals and queries. A signal delivers an external event (manager approved, rider cancelled) to a waiting workflow, which can sleep for days at no cost. A query reads workflow state without blocking it.

Version and continue-as-new. workflow.patched("fraud-check-v2") keeps in-flight workflows on the old path while new ones get the change. Long-lived workflows (subscriptions) use continue-as-new so history stays bounded.

• Calling datetime.now() or an HTTP API inside workflow code: replay diverges.
• A workflow that loops forever without continue-as-new: unbounded history.
• Activities without timeouts, so one hung call stalls the workflow.

Day 4Conflict Resolution

Last-write-wins loses data. The highest timestamp wins, so of two concurrent writes one silently disappears, and clock skew can drop the truly latest one. Fine for a location, a profile picture or "last seen"; wrong for carts and counters.

Vector clocks detect concurrency. Each node keeps a counter per node, increments its own on every write and takes the max on receive. If neither clock is ≤ the other in every entry, the writes were concurrent: a real conflict.

CRDTs merge by construction. Merge is commutative, associative and idempotent, so replicas converge whatever the order. G-Counter sums per-node counts; PN-Counter pairs two; OR-Set tags each add so a concurrent add beats a remove.

Pick by data semantics. Latest is right: LWW. Need to detect and ask: vector clocks with siblings (Riak allow_mult). Must merge offline: CRDTs. Collaborative text: OT (Google Docs) or a sequence CRDT (Yjs, Automerge).

• A counter as an LWW value: concurrent increments vanish. Use a PN-Counter.
• A plain set for add and remove: concurrent operations resurrect or drop items. Use an OR-Set.
• Never garbage-collecting tombstones, so CRDT state grows forever.

Day 5Leader Election and Coordination

Timeouts can't tell slow from dead. A GC pause, slow network or clock skew makes a live leader look dead; a follower takes over and two leaders accept writes: split-brain. You can't guarantee both safety and liveness, so favour safety.

Fencing tokens stop zombie leaders. Every election issues a higher token. The leader sends it with each request and resources reject anything below the highest they've seen, so a leader waking from a pause can't corrupt data.

Raft in one breath. Followers expect heartbeats; on an election timeout one becomes a candidate, bumps the term and asks for votes; a majority makes it leader. At most one leader per term, and terms act as fencing tokens.

Don't build it: ZooKeeper or etcd. ZooKeeper's ephemeral sequential znodes make the lowest sequence number the leader, deleted when its session expires; watches announce changes. etcd gives leases on Raft.

Epochs everywhere. Kafka elects partition leaders from the in-sync replicas and bumps an epoch so an old leader is rejected; Redis Sentinel needs a quorum to declare the master down; Kubernetes controllers hold a Lease object.

• A lock with no fencing token: a paused holder wakes up and writes over the new holder.
• "No heartbeat in 5 s, so I'm leader" without asking a quorum.
• Leases on wall-clock time; use monotonic clocks and renew early.

BossMultiplayer Game Matchmaking System

Size it with Little's law. 2M concurrent players, 1.6M in 5v5 matches (160,000 matches of 20 minutes): 8,000 matches start a minute, ~1,300 queue joins a second. At a 30 s wait, only ~40,000 players sit in queue.

One matchmaker per region, fenced. Two matchmakers reading the same queue would put the same 10 players in two matches. Elect one leader per region in etcd, and make every queue change carry its fencing token, checked atomically in Redis.

Ratings need strong consistency. Write ratings to the primary; a player viewing their own rating gets read-your-writes; the queue snapshots the rating from the primary at join time; leaderboards read cached replicas.

Match creation is a saga. Remove players from the queue, ready check, allocate a server, write the match, notify. If allocation fails, compensate by re-queuing confirmed players with priority; a leader crash leaves pending_match_id records to clean up.

Party state as an OR-Set. Kicks, leaves and invites can happen concurrently on different devices. With an OR-Set the merge is deterministic: concurrent removes of Carol both take effect, Dave's add survives. Parties match on their highest-rated player.

• Two matchmakers without fencing: players double-booked into two games.
• Matching on a stale replica's rating right after a match, creating unfair games.
• Removing players from the queue with no compensation if server allocation fails.