Amazon Prime Day Checkout.

What Does It Take to Never Lose an "Add to Cart"?

It is the first morning of Prime Day, 23 June 2026, in Seattle. Ishaan Mehta, a second-year master's student, has had the same cart for a week: a 27-inch monitor, a mechanical keyboard, and a phone case he has already decided against. On the 8:10 bus he finds a deal on noise-cancelling headphones and adds them from his phone. The bus goes under the freeway, his signal drops to one bar, and the app shows "Added to Cart" anyway.

At home, his laptop still has yesterday's cart open in a browser tab. He removes the phone case there, reviews the order, and presses "Place your order". The page spins for nine seconds. He presses it again. Somewhere behind those two presses, a cart written from two devices has to come out as exactly the three items he wants, the headphones have to be reserved before the deal's stock runs out, his card has to be charged once, and a warehouse in a different state has to learn about one order, not two. Millions of other people are doing something similar in the same minute.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a large e-commerce company. The interviewer shares two screenshots from the same morning: a cart on a phone reading "Subtotal (4 items)", and an order confirmation on a laptop reading "Order placed, thank you! · 3 items · Arriving Thursday".

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Orders and Cart Traffic#

The Hot Spots#

Key Metrics Summary#

System Architecture#

%% title: Cart, checkout and orders, in cells, on a shared catalogue and stock
flowchart TD
  subgraph L1["Layer 1 · Customers"]
    direction LR
    APP@{ icon: "sd:phone", label: "Shopping app<br/>cart, checkout" }
    WEB@{ icon: "sd:browser", label: "Website<br/>same APIs" }
  end
  subgraph L2["Layer 2 · Edge"]
    direction LR
    CDN@{ icon: "sd:cdn", label: "CDN<br/>images, scripts" }
    GW@{ icon: "sd:gateway", label: "Front door<br/>auth, priority, shedding" }
    RT@{ icon: "sd:router", label: "Cell router<br/>customer id to cell" }
  end
  subgraph L3["Layer 3 · One cell of many"]
    direction LR
    CART["Cart service<br/>always writable"]:::core
    CHK["Checkout and order<br/>idempotent placement"]:::hot
    CDB@{ icon: "sd:database", label: "Cell tables<br/>carts, orders, tokens" }
  end
  subgraph L4["Layer 4 · Shared, shuffle-sharded"]
    direction LR
    PRICE@{ icon: "sd:cache", label: "Price and promo<br/>read-mostly" }
    INV["Inventory<br/>per product and FC"]:::core
  end
  subgraph L5["Layer 5 · After the order"]
    direction LR
    Q@{ icon: "sd:queue", label: "Order queues<br/>per cell" }
    PAY@{ icon: "sd:bank", label: "Payment processors<br/>authorise, capture" }
    FC@{ icon: "sd:warehouse", label: "Fulfilment centres<br/>pick, pack, ship" }
  end
  APP ==> CDN
  APP ==> GW
  WEB ==> GW
  GW ==> RT
  RT ==> CART
  RT ==> CHK
  CART --> CDB
  CHK --> CDB
  CHK --> PRICE
  CHK ==>|"reserve"| INV
  CDB -.->|"order events"| Q
  Q ==> PAY
  Q ==> FC
  class APP,WEB client
  class CDN,GW,RT edge
  class CDB store
  class PRICE cache
  class Q queue
  class PAY external
  class FC external

KEY INSIGHTS:

• Two consistency regimes. The cart chooses availability and merges; order placement chooses correctness and uses atomic conditional writes. Mixing them is the classic mistake in both directions
• Cells by customer. Cart, checkout and orders are customer-scoped, so a customer id is a natural partition key for cells; a cell failure is a percentage, not an outage
• Shared services are few and shuffle-sharded. Prices and stock are scoped by product, not customer, so they can't live in customer cells; they're shared, read-heavy where possible, and isolated by shuffle sharding
• The page doesn't wait for the bank. Placement reserves stock and records the order; payment and fulfilment run behind a queue

Data Flow: Ishaan Places His Order#

%% title: Two presses of Place your order, one order
sequenceDiagram
  autonumber
  participant A as Ishaan's laptop
  participant R as Cell router
  participant C as Checkout
  participant P as Price and promo
  participant I as Inventory
  participant D as Cell tables
  A->>R: Place order, purchase id p-7f3, cart version 41
  R->>C: Cell 17, admitted at tier 1
  C->>D: Seen p-7f3?
  D-->>C: No
  C->>P: Re-price 3 lines
  P-->>C: Total matches the page
  C->>I: Reserve 3 lines, key p-7f3
  I-->>C: Reserved at FC SEA-4 and FC PDX-2
  C->>D: Order o-88 and token p-7f3, one transaction
  Note over A,C: The response is lost in a slow network
  A->>R: Place order again, purchase id p-7f3
  R->>C: Cell 17
  C->>D: Seen p-7f3?
  D-->>C: Yes, order o-88
  C-->>A: Order placed, o-88

MEANWHILE, IN PARALLEL:

• The transaction that wrote order o-88 also wrote an event to the cell's outbox; a relay puts it on the cell's order queue
• The order pipeline authorises Ishaan's card, marks the order confirmed, and releases one shipment to each fulfilment centre, each step keyed by the order id

Deep Dive 1: The Cart That Never Says No (Week 5 Concepts: Conflict Resolution, Consistency Models; Week 1: Replication Trade-offs)#

This builds on Week 5, Day 4 · Conflict Resolution, Week 5, Day 1 · Consistency Models and Week 1, Day 2 · Replication Trade-offs.

What is published: the Dynamo paper (SOSP 2007) describes a store where "the result of each modification" is "a new and immutable version of the data", tracked with vector clocks, lists of (node, counter) pairs. When two versions have no causal order, "both versions of the data must be kept and presented to a client (upon a read) for semantic reconciliation"; the cart "can choose to 'merge' the conflicting versions and return a single unified shopping cart". Both add and remove are writes; if the latest version isn't reachable, "the item is added to (or removed from) the older version and the divergent versions are reconciled later". Data is placed by consistent hashing with virtual nodes and replicated to N nodes; the common configuration was (N, R, W) = (3, 2, 2). Reads and writes use a "sloppy quorum" of "the first N healthy nodes", and a replica meant for a failed node is held elsewhere with a hint and handed back later; Merkle trees find replicas that drifted. Profiled for 24 hours, 99.94% of requests to the cart service saw exactly one version, 0.00057% two, 0.00047% three and 0.00009% four; the growth in divergent versions came "not by failures but due to the increase in number of concurrent writers", "usually triggered by busy robots (automated client programs) and rarely by humans". Dynamo answered 99.9995% of requests without timing out, with no data loss "to date".

%% title: Two writes from two devices, and a merge that keeps both
flowchart TD
  V1["Cart v1 · clock Sx 2<br/>monitor, keyboard, case"]:::service
  PH["Phone adds headphones<br/>coordinated by Sx · Sx 3"]:::core
  LT["Laptop removes case<br/>coordinated by Sz · Sx 2, Sz 1"]:::core
  READ{"Read finds two versions<br/>clocks not ordered"}
  MERGE["Merge by tags<br/>add tags kept, observed tags removed"]:::ok
  V4["Merged cart · Sx 4, Sz 1<br/>monitor, keyboard, headphones"]:::ok
  V1 ==> PH
  V1 ==> LT
  PH ==> READ
  LT ==> READ
  READ ==> MERGE ==> V4

What we'd build today. Amazon hasn't published how today's cart is stored. Our design keeps Dynamo's promise and fixes its known wart:

• The cart is a set of lines, not a blob. Each add creates a line with a unique tag (device id plus a counter); quantity changes and removes name the tags they saw. This is an observed-remove set, a standard conflict-free replicated data type: merging two versions is a union of tags minus the removed tags, and it's the same whatever order the merges happen in.
• Removes leave tombstones. A removed tag is remembered (with a timestamp) long enough for any lagging replica or offline device to catch up, then garbage-collected. That's what stops the phone case resurfacing.
• Quantities merge by intent. "Set quantity to 2" on one device and "set quantity to 3" on another is a real conflict; take the higher quantity for the same tag and show a banner ("We updated the quantity of 1 item"). Losing a sale is worse than asking.
• Single-writer in the common case. The cart is keyed by customer id. On a store with strongly consistent, conditional writes on one key (as DynamoDB offers), the normal path is a conditional update on the cart's version: no divergence at all. The merge is the fallback for the rare cases: an offline device syncing, a region failover, a guest cart merging at sign-in.
• Never refuse the add. If the cart's partition can't be reached within its latency budget, the add goes to a local, durable add-log in the same cell (another table, another partition) and is merged into the cart on the next read. The customer sees "Added to Cart" either way.

# cart/merge.py

"""
Merging divergent versions of a cart (illustrative). Each add creates a line with a
unique tag; a remove deletes only the tags it observed. Merging is a union of lines
minus the union of removed tags, so concurrent adds survive and removed items stay
removed. Quantities for the same tag merge to the larger value.
"""

from dataclasses import dataclass, field


@dataclass
class Cart:
    lines: dict = field(default_factory=dict)     # tag -> {"sku": str, "qty": int}
    removed: dict = field(default_factory=dict)   # tag -> removed_at (tombstones)


def add(cart: Cart, sku: str, qty: int, device: str, counter: int) -> str:
    tag = f"{device}:{counter}"                    # unique per device, never reused
    cart.lines[tag] = {"sku": sku, "qty": qty}
    return tag


def remove_sku(cart: Cart, sku: str, now: float) -> None:
    for tag, line in list(cart.lines.items()):    # only the tags this device has seen
        if line["sku"] == sku:
            del cart.lines[tag]
            cart.removed[tag] = now


def merge(a: Cart, b: Cart) -> Cart:
    removed = {**a.removed, **b.removed}
    lines = {}
    for tag in (a.lines.keys() | b.lines.keys()) - removed.keys():
        qa = a.lines.get(tag, {}).get("qty", 0)
        qb = b.lines.get(tag, {}).get("qty", 0)
        lines[tag] = {"sku": (a.lines.get(tag) or b.lines[tag])["sku"], "qty": max(qa, qb)}
    return Cart(lines=lines, removed=removed)


def view(cart: Cart) -> dict:
    out = {}
    for line in cart.lines.values():               # two tags for one SKU: show one line
        out[line["sku"]] = out.get(line["sku"], 0) + line["qty"]
    return out

Deep Dive 2: From Dynamo to DynamoDB: Partitions, Hot Keys and the Peak (Week 1 Concepts: Partitioning, Hot Keys; Week 10: Capacity Planning)#

This builds on Week 1, Day 1 · Partitioning Deep Dive, Week 1, Day 4 · Hot Keys and Skew and Week 10, Day 4 · Capacity Planning.

What is published: the DynamoDB paper (USENIX ATC 2022) says Dynamo "was created in response to the need for a highly scalable, available, and durable key-value database for shopping cart data", was "a single-tenant system", and that "teams were responsible for managing their own Dynamo installations", which "became a barrier to adoption". DynamoDB, launched in 2012, splits each table into partitions, each replicated across Availability Zones; a partition's replicas use Multi-Paxos, only the leader serves writes and strongly consistent reads, and a write is acknowledged "once a quorum of peers persists the log record". A healthy write quorum is "two out of the three replicas from different AZs". Every write can carry a condition, and multi-item ACID transactions are supported. Early on, throughput was divided evenly across a table's partitions, which caused "hot partitions" and "throughput dilution"; bursting let partitions use capacity unused in the last 300 seconds, and adaptive capacity "eliminated over 99.99% of the throttling due to skewed access pattern". It was replaced by global admission control, token buckets tracked centrally and handed to request routers, plus splitting for consumption, where a hot partition is split at a point chosen from its observed keys; splits "usually complete in the order of minutes", but a partition "receiving high traffic to a single item" won't benefit. On-demand tables accommodate "up to double the previous peak traffic on the table". For Prime Day 2021 the paper reports "trillions of API calls" from Amazon's systems over 66 hours, peaking at 89.2 million requests a second.

%% title: One write to DynamoDB, as the 2022 paper describes it
flowchart TD
  APP["Cart or order service<br/>conditional write"]:::service
  RR@{ icon: "sd:router", label: "Request router<br/>auth, local token bucket" }
  GAC@{ icon: "sd:servers", label: "Global admission control<br/>vends tokens every few seconds" }
  MD@{ icon: "sd:index", label: "Partition map<br/>MemDS, refreshed constantly" }
  LEAD@{ icon: "sd:database", label: "Partition leader · AZ a<br/>writes, consistent reads" }
  F2@{ icon: "sd:database", label: "Replica · AZ b" }
  F3@{ icon: "sd:database", label: "Replica or log replica · AZ c" }
  APP ==> RR
  RR -.-> GAC
  RR -.-> MD
  RR ==> LEAD
  LEAD ==>|"log record, quorum 2 of 3"| F2
  LEAD --> F3
  class RR edge
  class GAC,MD service
  class LEAD,F2,F3 store

• Pick keys that spread. Carts by customer id, orders by order id (with a customer-to-orders index), idempotency records by customer plus purchase id. Every one of these is high-cardinality and evenly used; none is hot by design.
• Name the hot keys and design them away. The only keys everyone touches are a limited deal's stock and the counters behind "% claimed". Shard them (Deep Dive 5). A partition split can't help a single item, and a bigger table can't either.
• Pre-warm by rehearsal. The on-demand rule is relative to a table's own history. Each table's Prime Day peak is forecast per cell; anything above twice its previous peak is driven there in a load test before the day.
• Use conditions, not read-then-write. Cart version checks, idempotency records and stock decrements are all conditional writes: the store decides atomically, so no service needs a lock.
• Predictability over efficiency. The paper's lesson: "do not allow [caches] to hide the work that would be performed in their absence". If the price cache's hit rate drops on the day, the price service behind it must still be sized for the traffic it will see, or the cache's failure becomes the service's failure.

# capacity/prewarm.py

"""
Which tables need rehearsal before the peak (illustrative). An on-demand table
absorbs up to twice its previous peak at once; above that it splits partitions,
which takes minutes. Anything forecast above 2x its own history is load-tested to
its forecast first. A single hot item is flagged for write sharding instead.
"""

PARTITION_WRITES = 1000          # the DynamoDB paper's example per-partition maximum


def plan(tables: list) -> list:
    actions = []
    for t in tables:              # {"name", "previous_peak_wps", "forecast_wps", "hottest_item_wps"}
        if t["hottest_item_wps"] > 0.5 * PARTITION_WRITES:
            shards = -(-t["hottest_item_wps"] * 2 // PARTITION_WRITES)
            actions.append((t["name"], f"shard hottest item into >= {shards} keys"))
        if t["forecast_wps"] > 2 * t["previous_peak_wps"]:
            actions.append((t["name"], f"rehearse at {t['forecast_wps']} writes/s before the day"))
    return actions

Deep Dive 3: Cells and Shuffle Sharding: Making Every Failure Small (Week 9 Concepts: Tenant Isolation, Noisy Neighbours; Week 10: Deployment Strategies)#

This builds on Week 9, Day 1 · Tenant Isolation, Week 9, Day 2 · Noisy Neighbour and Week 10, Day 3 · Deployment Strategies.

What is published: AWS's guidance Reducing the Scope of Impact with Cell-Based Architecture (September 2023) says that "for more than a decade, our service teams have used cell-based architecture". A cell is an independent instance of a workload that "does not share state with other cells"; a thin routing layer sends each request to a cell by a partition key such as a customer id, chosen to "match the grain of the service"; cells have a fixed maximum size and grow by adding cells, so each can be "stress tested and pushed past their breaking point"; and "if a workload uses 10 cells to service 100 requests, when a failure occurs in one cell, 90% of the overall requests would be unaffected". The Builders' Library article Workload isolation using shuffle-sharding (2019) explains the idea with 8 workers: plain sharding into 4 pairs limits an attack to 25%; assigning each customer a random pair of the 8 gives 28 possible "shuffle shards", so a problem reaches "just 1/28th". Route 53 uses 2,048 virtual name servers and gives each domain 4 of them: "730 billion possible shuffle shards", and no domain shares "more than two virtual name servers with any other customer domain".

%% title: Customers in cells; product-scoped services shuffle-sharded
flowchart TD
  RT@{ icon: "sd:router", label: "Cell router<br/>map in memory, from a file" }
  C1["Cell 1<br/>cart, checkout, orders, tables"]:::core
  C17["Cell 17 · Ishaan<br/>cart, checkout, orders, tables"]:::hot
  C40["Cell 40<br/>cart, checkout, orders, tables"]:::core
  SS@{ icon: "sd:hexagons", label: "Price service · 32 groups<br/>each cell uses 4" }
  INV@{ icon: "sd:database", label: "Inventory<br/>partitioned by product and FC" }
  CP@{ icon: "sd:dashboard", label: "Control plane<br/>place, deploy, migrate cells" }
  RT ==> C1
  RT ==> C17
  RT ==> C40
  C17 ==> SS
  C17 ==> INV
  CP -.->|"one cell at a time"| C17
  class RT edge
  class SS service
  class INV store
  class CP service

What we'd build (our design; Amazon hasn't published its retail cells):

• Partition key: customer id. Cart, checkout, orders and idempotency records are all scoped to one customer, the grain of the service. A cell holds a fixed maximum, say 8 million customers (assumed); ~300 million active carts (also assumed) → ~40 cells. At the design peak, each cell takes ~10,000 cart operations and ~500 orders a second.
• The router is dumb and static. It hashes the customer id to a cell from a mapping file loaded into memory. If the control plane is down, routing still works from the last mapping: the static stability the guidance asks for ("the data plane to continue operating even if the control plane is down").
• Deploys go cell by cell. A new build goes to one cell, bakes, then to a few, then to the rest, with automatic rollback on the cell's own alarms. A bad build is a 2.5% problem for minutes, not a 100% problem.
• Cells are tested to break. Because every cell is the same size, the load test for Prime Day is a test of one cell pushed past its forecast, not a guess about the whole fleet.
• Shared services get shuffle shards. Price and promotion lookups and the inventory service's request handlers are product-scoped, so they can't live in customer cells. Each cell is assigned its own combination of 4 of 32 server groups, so a poison request or a flood from one cell can't take all the capacity of any other cell.

# cells/shards.py

"""
Cell routing and shuffle-shard assignment (illustrative). Customers map to cells by
a hash over a static table loaded at start-up; each cell gets its own combination of
shared server groups, with no two cells sharing more than two groups.
"""

import hashlib
import itertools
import random

CELLS = 40
GROUPS = 32          # price service server groups
PER_CELL = 4


def cell_of(customer_id: str, mapping: list) -> int:
    h = int(hashlib.sha256(customer_id.encode()).hexdigest(), 16)
    return mapping[h % len(mapping)]          # mapping: slots -> cell, from the control plane


def assign_shuffle_shards(seed: int = 7) -> dict:
    rng = random.Random(seed)
    combos = list(itertools.combinations(range(GROUPS), PER_CELL))   # 35,960 of them
    rng.shuffle(combos)
    chosen = {}
    for combo in combos:
        if all(len(set(combo) & set(other)) <= 2 for other in chosen.values()):
            chosen[len(chosen)] = combo
            if len(chosen) == CELLS:
                break
    return chosen                              # cell -> 4 groups, overlap <= 2 with any cell

Deep Dive 4: Placing the Order Exactly Once (Week 2 Concepts: Idempotency, Timeout Hell; Week 3: Transactional Outbox)#

This builds on Week 2, Day 2 · Idempotency in Practice, Week 2, Day 1 · Timeout Hell and Week 3, Day 2 · Transactional Outbox.

What is published: the Builders' Library article Making retries safe with idempotent APIs (Malcolm Featonby) describes Amazon's "preferred approach": "a unique caller-provided client request identifier" in the API contract (the ClientToken in EC2's RunInstances). On a first request, the service records the identifier and does the work; "the process that combines recording the idempotent token and all mutating operations related to servicing the request must meet the properties for an atomic, consistent, isolated, and durable (ACID) operation". A repeat returns "a semantically equivalent response", rather than a "ResourceAlreadyExists" code that leaves the caller unsure who created it. If the same token arrives with different parameters, "we return a validation error indicating a parameter mismatch", so the parameters are stored with the token. Retention for EC2 is "the lifetime of the resource, plus an interval" after which late retries are no longer plausible. Marc Brooker's Timeouts, retries, and backoff with jitter adds the other half: retries are "selfish"; a five-deep call stack with three tries at each layer multiplies load on the database "243x", so Amazon's practice is "to retry at a single point in the stack", with capped exponential backoff, jitter, and a token bucket limiting retries, which AWS added to its SDK in 2016. Timeouts are set from the dependency's latency at a chosen false-timeout rate, for example its p99.9 for 0.1%.

%% title: Place order with a purchase id, a retry, and a mismatched replay
sequenceDiagram
  autonumber
  participant A as App
  participant C as Checkout
  participant I as Inventory
  participant D as Cell tables
  A->>C: Place order, id p-7f3, cart hash h1
  C->>I: Reserve lines, key p-7f3
  I-->>C: Reserved
  C->>D: Transaction · put token p-7f3 if absent · put order o-88 · put outbox row
  D-->>C: Committed
  Note over A,C: Response lost, app retries after 3 s with jitter
  A->>C: Place order, id p-7f3, cart hash h1
  C->>D: Get token p-7f3
  D-->>C: Order o-88, cart hash h1
  C-->>A: Order placed, o-88
  A->>C: Place order, id p-7f3, cart hash h2
  C->>D: Get token p-7f3
  D-->>C: Order o-88, cart hash h1
  C-->>A: 409 parameter mismatch, review your cart

• The id comes from the page, not the click. Generating it at render time makes a double-press and an app retry the same intent. Generating it per click would turn a double-press into two orders.
• One transaction, three items. The token record (conditional: only if absent), the order, and an outbox row for the pipeline commit together. DynamoDB's ACID transactions across items in one table or several make this a single call. A crash leaves either all three or none.
• Reservation is keyed too. The inventory reservation carries the purchase id, so a retry that reached inventory twice still holds one set of units. If the order transaction fails, the reservation is released by the same key.
• A race between two retries. Two copies of the same request can both miss the token and both reserve; only one transaction's "put if absent" succeeds. The loser reads the winner's order, releases nothing (its reservation key is the same), and returns the winner's response.
• Retry once, at the edge. The app retries with jitter; checkout does not retry its own downstream calls more than once, and inventory does not retry at all. Three layers of retries are how one slow partition becomes 27× the load.

# checkout/place_order.py

"""
Placing an order exactly once (illustrative). The purchase id comes from the
rendered checkout page. The token, the order and an outbox row commit in one
transaction; a retry returns the same order, and a replay with different contents
is rejected as a mismatch.
"""

import hashlib
import json


class Mismatch(Exception):
    pass


def cart_hash(lines: list, total_cents: int, address_id: str) -> str:
    body = json.dumps({"l": sorted(lines), "t": total_cents, "a": address_id})
    return hashlib.sha256(body.encode()).hexdigest()[:16]


def place_order(db, inventory, customer: str, purchase_id: str, lines: list,
                total_cents: int, address_id: str) -> dict:
    h = cart_hash(lines, total_cents, address_id)
    key = f"{customer}#{purchase_id}"
    seen = db.get("tokens", key)
    if seen:
        if seen["hash"] != h:
            raise Mismatch(purchase_id)              # same id, different intent
        return {"order_id": seen["order_id"], "status": "PLACED"}

    held = inventory.reserve(lines, key=key)          # idempotent by key
    order_id = f"o-{hashlib.sha256(key.encode()).hexdigest()[:10]}"
    ok = db.transact([
        ("put_if_absent", "tokens", key, {"order_id": order_id, "hash": h}),
        ("put", "orders", order_id, {"customer": customer, "lines": lines,
                                     "fcs": held["fcs"], "state": "PLACED"}),
        ("put", "outbox", order_id, {"event": "OrderPlaced"}),
    ])
    if not ok:                                        # another copy of this request won
        seen = db.get("tokens", key)
        if seen["hash"] != h:
            inventory.release(key)
            raise Mismatch(purchase_id)
        return {"order_id": seen["order_id"], "status": "PLACED"}
    return {"order_id": order_id, "status": "PLACED"}

Deep Dive 5: Reserving Stock Without Overselling (Week 5 Concepts: Consistency Models; Week 1: Hot Keys and Skew)#

This builds on Week 5, Day 1 · Consistency Models and Week 1, Day 4 · Hot Keys and Skew.

What is published: Amazon has not published how it reserves stock or runs limited-quantity deals. What is published are the building blocks: DynamoDB's conditional writes and transactions, its note that a single hot item can't be split away (DynamoDB paper, 2022), and that DynamoDB serves "all Amazon fulfillment centers" (AWS, Prime Day posts). The design in this deep dive is ours. Flipkart's sale (Bonus Problem 27) adds a waiting line in front of a deal; here we show the counter design underneath.

%% title: Units of stock for one deal at one fulfilment centre
stateDiagram-v2
  [*] --> AVAILABLE: received at the FC
  AVAILABLE --> RESERVED: order placed, key p-7f3
  RESERVED --> AVAILABLE: order failed or cancelled, released by key
  RESERVED --> ALLOCATED: payment authorised
  ALLOCATED --> PICKED: released to the warehouse
  PICKED --> SHIPPED: left the building
  ALLOCATED --> AVAILABLE: order cancelled before pick
  SHIPPED --> [*]
  class SHIPPED ok
  class RESERVED queue

• One row per product per fulfilment centre, with on_hand, reserved and allocated; available = on_hand − reserved − allocated. Every move is a conditional update on that row; a reconciler checks the arithmetic against warehouse counts.
• Reservations are records, keyed by the purchase id, with the lines and centres they hold. A retry finds the record; a release names it. No reservation is anonymous.
• No clock-based expiry for placed orders. A reservation belongs to an order, and the order's state machine releases it (Deep Dive 6). A sweeper only cleans reservations whose order transaction never committed (the crash between reserve and commit), after a few minutes.
• Sub-counters only where needed. Ordinary products have one row per centre and a few writes a second. Only products forecast to be hot (deals, launches) are split, before the deal goes live; when few units remain, a sweeper gathers the stragglers into one sub-counter so the last units still sell.
• Multi-line orders reserve together. The monitor, keyboard and headphones are reserved in one transaction across their rows (DynamoDB transactions cover multiple items atomically). If the headphones are gone, nothing is held, and the page says which item is sold out.

# inventory/reserve.py

"""
Reserving deal stock from split counters (illustrative). Each sub-counter can only
be decremented while it has stock, and the sub-counters sum to the deal quantity,
so the deal cannot oversell. The reservation is a record keyed by the purchase id.
"""

import random

TRIES = 3


class SoldOut(Exception):
    pass


def reserve_deal_unit(db, deal: str, fcs: list, key: str) -> dict:
    existing = db.get("reservations", key)
    if existing:
        return existing                                     # a retry: same units
    for fc in fcs:                                          # nearest centre first
        subs = db.query("deal_stock", prefix=f"{deal}#{fc}#")
        live = [s["id"] for s in subs if s["left"] > 0]
        for sub in random.sample(live, min(TRIES, len(live))):
            ok = db.transact([
                ("update_if", "deal_stock", sub, {"left": -1}, "left >= 1"),
                ("put_if_absent", "reservations", key,
                 {"deal": deal, "fc": fc, "sub": sub, "state": "RESERVED"}),
            ])
            if ok:
                return db.get("reservations", key)
    raise SoldOut(deal)


def release(db, key: str) -> None:
    r = db.get("reservations", key)
    if r and r["state"] == "RESERVED":
        db.transact([
            ("update_if", "reservations", key, {"state": "RELEASED"}, "state = RESERVED"),
            ("update", "deal_stock", r["sub"], {"left": +1}),
        ])

Deep Dive 6: The Order Pipeline After "Place Your Order" (Week 3 Concepts: Queues, Backpressure; Week 5: Saga Orchestration)#

This builds on Week 3, Day 1 · Queue vs Stream, Week 3, Day 3 · Backpressure and Flow Control, Week 5, Day 3 · Saga Orchestration and Week 3, Day 4 · Dead Letters and Poison Pills.

What is published: the Builders' Library article Avoiding insurmountable queue backlogs uses "fulfilling orders placed on amazon.com" as an example of a workflow built on queues, and warns that "queueing that is meant to improve availability can backfire" by "dramatically" increasing recovery time. Its practices: measure the age of messages, not just the count; separate workloads into separate queues, or shuffle-shard customers across a fixed set of queues as Lambda does; prefer "LIFO-ish behavior" for fresh work and "sideline old traffic" into a backlog queue; and protect each layer, since "there will always be some workload that gets around the front-door admission control". Using load shedding to avoid overload adds that prioritisation should happen "as early as possible (such as in the service that receives the initial request from the client)", that a request for a search crawler is "likely to be less critical to serve than a request that originates from a human", and that servers should drop work whose client has already timed out. Amazon has not published its order pipeline's design; what follows is ours.

%% title: One order, from placed to released, every step keyed by the order id
flowchart TD
  OB@{ icon: "sd:log", label: "Outbox row<br/>written with the order" }
  LIVE@{ icon: "sd:queue", label: "Live queue · cell 17<br/>fresh orders first" }
  SIDE@{ icon: "sd:queue", label: "Sideline queue<br/>old orders after a backlog" }
  ORCH["Order orchestrator<br/>state machine per order"]:::hot
  PAY@{ icon: "sd:card", label: "Payment authorisation<br/>key o-88, timeout from p99.9" }
  REL["Release to fulfilment<br/>one shipment per FC"]:::core
  COMP["Compensate<br/>release stock, notify"]:::bad
  OB ==> LIVE
  LIVE ==> ORCH
  SIDE -.-> ORCH
  ORCH ==> PAY
  PAY ==>|"authorised"| REL
  PAY -.->|"declined for good"| COMP
  class OB store
  class LIVE,SIDE queue
  class PAY external

• An orchestrated saga, not a chain of events. One orchestrator per order holds its state (PLACED → AUTHORISED → CONFIRMED → RELEASED, or PAYMENT_PENDING_CUSTOMER → CANCELLED) and calls each step with the order id as the idempotency key. A crash resumes from the stored state; a repeated step is a no-op.
• Authorise now, charge later. Placement records the order; the pipeline authorises the card (a hold), and the charge is captured when the shipment leaves the fulfilment centre. That's the card networks' authorise-then-capture model (Bonus Problem 4), and it means a cancelled-before-shipping order costs a released hold, not a refund. This is our design choice; Amazon's own charge timing isn't covered by the sources here.
• A timeout is "unknown", not "failed". If the processor doesn't answer within its timeout, the step is retried later with the same key, and never as a new authorisation. A decline after retries moves the order to PAYMENT_PENDING_CUSTOMER and emails the customer; stock stays reserved for a fixed number of days, then the order is cancelled and released.
• Queues per cell, ages on the dashboard. Each cell has its own live queue, so one cell's backlog doesn't delay another's orders. The alarm is the age of the oldest message in the live queue, not the queue length.
• Bulkheads per processor. Each payment processor gets its own worker pool and concurrency limit. A slow processor fills its own pool and its orders wait; orders for other processors flow.
• Release once. The hand-off to a fulfilment centre is a write keyed by (order id, centre). The centre's system ignores a second copy. A shipment can be cancelled until it's picked; after that, the customer's cancellation becomes a return.

# orders/orchestrator.py

"""
Advancing one order (illustrative). Each step is called with the order id as its
idempotency key, so a crash or redelivery repeats nothing. A processor timeout is
"unknown" and retried later with the same key; a final decline waits for the
customer, then cancels and releases stock.
"""

import random

PENDING_CUSTOMER_DAYS = 3


def advance(db, payments, fulfilment, inventory, order_id: str, now: float) -> str:
    o = db.get("orders", order_id)
    if o["state"] == "PLACED":
        r = payments.authorise(order_id, o["payment_method"], o["total_cents"], key=order_id)
        if r == "AUTHORISED":
            return db.cas("orders", order_id, "PLACED", "AUTHORISED")
        if r == "UNKNOWN":
            return db.schedule(order_id, at=now + 30 + random.uniform(0, 30))   # backoff, jitter
        db.cas("orders", order_id, "PLACED", "PAYMENT_PENDING_CUSTOMER",
               deadline=now + PENDING_CUSTOMER_DAYS * 86400)
        return "EMAIL_CUSTOMER"
    if o["state"] == "AUTHORISED":
        for fc in o["fcs"]:
            fulfilment.release(order_id, fc, key=f"{order_id}#{fc}")             # once per FC
        return db.cas("orders", order_id, "AUTHORISED", "RELEASED")
    if o["state"] == "PAYMENT_PENDING_CUSTOMER" and now > o["deadline"]:
        inventory.release(o["reservation_key"])
        return db.cas("orders", order_id, "PAYMENT_PENDING_CUSTOMER", "CANCELLED")
    return o["state"]

Ishaan's Prime Day, From the Platform's Side#

Edge Cases#

Monitoring Dashboard#

SLOs for the Sale#

The interviewer leans back. "One more."

Summary: Concepts Applied from 10-Week Course#

Why Amazon Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

The cart:

Placing the order:

Surviving the day:

Sources#

Papers:

• Paper

  Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and Werner Vogels — Dynamo: Amazon's Highly Available Key-value Store (cart "always writeable"; disks failing, routes flapping, "tornados"; "Add to Cart" never forgotten or rejected; merge and resurfacing deletes; vector clocks; (3,2,2); sloppy quorum and hinted handoff; Merkle trees; 150+ services per page; 99.9th-percentile SLAs and the 300 ms example; tens of millions of requests and well over 3 million checkouts in a day; 99.94% one version over 24 hours; divergence from concurrent writers, "busy robots"; 99.9995% success), SOSP, October 2007: https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

• Paper

  Mostafa Elhemali, Niall Gallagher, Nicholas Gordon, Joseph Idziorek, Richard Krog, Colin Lazier, Erben Mo, Akhilesh Mritunjai, Somu Perianayagam, Tim Rath, Swami Sivasubramanian, James Christopher Sorenson III, Sroaj Sosothikul, Doug Terry and Akshat Vig — Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service (Dynamo created for shopping-cart data, single-tenant; DynamoDB 2012; partitions, Multi-Paxos, two-of-three AZ write quorum, log replicas; conditional writes and ACID transactions; 1,000-WCU partition example; bursting, adaptive capacity eliminating over 99.99% of skew throttling, global admission control, split for consumption in minutes, not for a single hot item; on-demand up to 2× the previous peak; lease takeover in a couple of seconds; Prime Day 2021: 66 hours, peak 89.2 million requests a second; "do not allow [caches] to hide the work"), USENIX ATC, July 2022: https://www.usenix.org/system/files/atc22-elhemali.pdf

AWS News Blog, Prime Day posts:

• Book

  Jeff Barr — Prime Day 2017 – Powered by AWS (30-hour event; DynamoDB 3.34 trillion requests, peak 12.9 million a second), 14 September 2017: https://aws.amazon.com/blogs/aws/prime-day-2017-powered-by-aws/

• Book

  Jeff Barr — Amazon Prime Day 2019 – Powered by AWS (48 hours; 7.11 trillion DynamoDB calls, peak 45.4 million a second; 442 fulfilment centres), 16 August 2019: https://aws.amazon.com/blogs/aws/amazon-prime-day-2019-powered-by-aws/

• Blog

  Jeff Barr — Amazon Prime Day 2022 – AWS for the Win! (DynamoDB peak 105.2 million a second; SQS 70.5 million messages a second; more than 100,000 items a minute), 25 July 2022: https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-for-the-win/

• Book

  Jeff Barr — Prime Day 2023 Powered by AWS – All the Numbers (DynamoDB peak 126 million a second; SQS 86 million messages a second), 2 August 2023: https://aws.amazon.com/blogs/aws/prime-day-2023-powered-by-aws-all-the-numbers/

• Blog

  Channy Yun and Jeff Barr — How AWS powered Prime Day 2024 for record-breaking sales (two-day event; DynamoDB peak 146 million a second; 733 fault-injection experiments), 13 August 2024: https://aws.amazon.com/blogs/aws/how-aws-powered-prime-day-2024-for-record-breaking-sales/

• Blog

  Channy Yun — AWS services scale to new heights for Prime Day 2025: key metrics and milestones (four-day event; DynamoDB peak 151 million a second; SQS 166 million messages a second; 6,800+ fault-injection experiments), 26 August 2025: https://aws.amazon.com/blogs/aws/aws-services-scale-to-new-heights-for-prime-day-2025-key-metrics-and-milestones/

• Book

  Channy Yun — All the numbers: Amazon Prime Day 2026 powered by AWS (23–26 June 2026; DynamoDB 59 trillion+ requests, peak 192 million a second; SQS peak 213 million messages a second; 44,000+ fault-injection experiments, more than six times 2025), 6 October 2026: https://aws.amazon.com/blogs/aws/all-the-numbers-amazon-prime-day-2026-powered-by-aws/

Amazon Builders' Library and AWS guidance:

• Docs

  Malcolm Featonby — Making retries safe with idempotent APIs (client request identifier; ACID recording of token and mutations; semantically equivalent responses; parameter-mismatch validation error; retention for the resource's lifetime plus an interval): https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

• Read

  Marc Brooker — Timeouts, retries, and backoff with jitter (timeouts from p99.9 at 0.1% false timeouts; retries are "selfish"; 243× at five layers; retry at a single point; token-bucket retry limits in the SDK since 2016; jitter), 2019: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

• Read

  Workload isolation using shuffle-sharding (8 workers, 28 shuffle shards, 1/28th impact; Route 53: 2,048 virtual name servers, 4 per domain, 730 billion shuffle shards, no more than two shared), 2019: https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/

• Read

  Avoiding insurmountable queue backlogs (order fulfilment as a queue workflow; double capacity for another hour after an hour's outage; age of messages; separate and shuffle-sharded queues; LIFO-ish behaviour; sidelining old traffic), 2019: https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/

• Read

  Using load shedding to avoid overload (prioritise early, at the service receiving the request; crawler traffic below human traffic; drop work whose client timed out; bounded work), 2019: https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/

• Read

  Static stability using Availability Zones (overprovision so the system keeps working without launching instances if a zone is impaired), 2019: https://aws.amazon.com/builders-library/static-stability-using-availability-zones/

• Book

  AWS Well-Architected — Reducing the Scope of Impact with Cell-Based Architecture (cells used by AWS service teams for more than a decade; no shared state; partition key matching the grain of the service; fixed maximum cell size; 10 cells, 90% unaffected; control plane and data plane; cell migration; "100% of customers ... 5% failure rate, or 5% of customers ... 100%"), 20 September 2023: https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html

Amazon executives and engineers:

• Read

  Werner Vogels — Amazon DynamoDB – a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications (outages at the height of the 2004 holiday season; Dynamo built for the shopping cart and session service), All Things Distributed, January 2012: https://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html

• Read

  Werner Vogels — A Decade of Dynamo (2004 Oracle limits; ~70% key-value and ~20% single-table operations; DynamoDB launched January 2012; Prime Day 2017 peak 12.9 million a second), All Things Distributed, 2 October 2017: https://www.allthingsdistributed.com/2017/10/a-decade-of-dynamo.html

• Blog

  Jeff Barr — Migration Complete – Amazon's Consumer Business Just Turned off its Final Oracle Database (75 PB in ~7,500 Oracle databases; teams including Ordering, Consumer Payments and Retail Systems; low-latency services to DynamoDB, transactional relational workloads to Aurora and RDS), AWS News Blog, 15 October 2019: https://aws.amazon.com/blogs/aws/migration-complete-amazons-consumer-business-just-turned-off-its-final-oracle-database/

Design note: the cart's tagged-line merge, the add-log fallback, the cells and their size and count, the shuffle-shard layout, the purchase id and its hash, the reservation and sub-counter design, the orchestrated order pipeline, authorise-then-capture, the timeouts, thresholds, dashboards and every number labelled as an assumption or estimate are an illustrative design for the problem, not a description of Amazon's internal systems beyond what Amazon and AWS have published. Order rates, cart operations per order, growth factors, deal sizes, buyer counts and the 2027 projection are assumptions. Ishaan Mehta and everything he sees are illustrative.

Further Reading#

Related systems to study:

• Side quest

  Flipkart Big Billion Days (Bonus Problem 27 in this series): a sale's opening minute, a waiting line for flash deals, stock holds and a payment surge through Indian banks

  ↗(opens in new tab)
• Side quest

  AWS S3 (Bonus Problem 8): another Amazon system built for durability at scale

  ↗(opens in new tab)
• Side quest

  Visa and Mastercard (Bonus Problem 4): authorise now, capture and settle later

  ↗(opens in new tab)
• Side quest

  BookMyShow (Bonus Problem 16): seats that must never be sold twice, under a flash crowd

  ↗(opens in new tab)

From the war stories:

• Read

  A brief network blip pushed DynamoDB's error rate to 55%, because its answers had crept up to a timeout

• Read

  AWS's hidden internal network jammed, and took its own monitoring down with it

• Read

  Flipkart's founders said sorry the day after the Big Billion Day

Foundational concepts (this course):

• Lesson

  Week 1, Day 1 — Partitioning Deep Dive

  ↗(opens in new tab)
• Lesson

  Week 1, Day 2 — Replication Trade-offs

  ↗(opens in new tab)
• Lesson

  Week 1, Day 4 — Hot Keys and Skew

  ↗(opens in new tab)
• Lesson

  Week 2, Day 1 — Timeout Hell

  ↗(opens in new tab)
• Lesson

  Week 2, Day 2 — Idempotency in Practice

  ↗(opens in new tab)
• Lesson

  Week 3, Day 1 — Queue vs Stream

  ↗(opens in new tab)
• Lesson

  Week 3, Day 2 — Transactional Outbox

  ↗(opens in new tab)
• Lesson

  Week 3, Day 3 — Backpressure and Flow Control

  ↗(opens in new tab)
• Lesson

  Week 3, Day 4 — Dead Letters and Poison Pills

  ↗(opens in new tab)
• Lesson

  Week 5, Day 1 — Consistency Models

  ↗(opens in new tab)
• Lesson

  Week 5, Day 3 — Saga Orchestration

  ↗(opens in new tab)
• Lesson

  Week 5, Day 4 — Conflict Resolution

  ↗(opens in new tab)
• Lesson

  Week 9, Day 1 — Tenant Isolation

  ↗(opens in new tab)
• Lesson

  Week 9, Day 2 — Noisy Neighbour

  ↗(opens in new tab)
• Lesson

  Week 10, Day 3 — Deployment Strategies

  ↗(opens in new tab)
• Lesson

  Week 10, Day 4 — Capacity Planning

  ↗(opens in new tab)

"Never refuse an add, never repeat an order, and never let one mistake reach every customer."