Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.
How Do You Take Hundreds of Thousands of Orders in the First Minute of the Day Without Losing One or Placing One Twice?
It is 9:14:58 on a Monday. Asha's buy order is typed in, limit price set, thumb over the button. Across India, several lakh other people are doing the same thing.
At 9:15:00 the market opens and they all tap at once. Asha's order has to be checked against her money, sent to the exchange over a leased line, acknowledged and shown back to her as "open", while the price on her screen keeps moving. If the confirmation takes three seconds she will tap again. The design has to make sure that second tap can't become a second order.
On a normal day that is a few hundred milliseconds of work. On a budget day or an election result day it is the same work at several times the volume, and the exchange itself may be slower to answer.
- 67.9 lakh — Zerodha's NSE active clients in September 2026 (68.93 lakh in March 2026, 78.88 lakh in March 2025)
- 34.5 million — orders on Kite on 4 June 2024, Lok Sabha election results day, with more than 8 million logins
- 9:14 to 9:20 — "an exceptional amount of activity", in Zerodha's 2019 postmortem
- 1 million+ — concurrent users "at any given moment" (Zerodha's tech blog, 2021)
- Tens of millions — market-data packets a second streamed by Kite's ticker (2021)
- 40 ms — Kite's client round-trip latency baseline over a good connection
For context:
Zerodha is one of India's largest stockbrokers; Kite is its trading platform
NSE's equity market opens for continuous trading at 9:15 am, after a pre-open call auction from 9:00
Everything not in the list above is an illustrative design target, not Zerodha's internal architecture
Nothing here is investment advice: the subject is the software
This is the system we'll design today.
Kite took 34.5 million orders in a day and peaks at 9:15:00: ~670 orders a second on average, 7,500 in the first minute, near 15,000 at the busiest second, while ~30 million price packets a second stream out. Orders pass gateway, order service, RMS and exchange gateway exactly once; reads come from a hot cache and prices from feed handlers to ticker servers. The deep dives cover idempotency and UNKNOWN orders, the in-memory RMS, tick coalescing, the open spike and failures, then Budget day with Little's law and the edge cases.
The Interview Begins#
You're interviewing at a company building a retail stockbroking platform for India. The engineering lead opens a graph of one day's order traffic. It is almost flat, except for a single tower at the very start.
"That spike is 9:15. Every morning. Design the platform: placing orders, checking margin, talking to the exchanges, streaming prices. Then tell me how it survives that tower."
Retail clients place, modify and cancel orders, watch live prices and their positions, and add or withdraw money. The broker checks every order against the client's funds, sends it to the exchange, and reports every state change back.
Requirements
- Never lose an order, and never send one order to the exchange twice
- Check margin exactly, before the order leaves, at open-bell volume
- Show the true state of every order, including "we don't know yet"
- Stream live prices to more than a million screens
- Keep every order event for regulators, for years
"Don't design the matching engine; that's the exchange's job. You're the broker: everything between the client's thumb and the exchange's door."
Phase 1: Requirements Clarification (5 minutes)#
"I'd like to pin down where our responsibility ends, and what 'correct' means for an order."
Your Questions#
"Do we run the matching engine?"
"No. NSE, BSE and MCX match orders. You connect as a member over dedicated leased lines, and receive their acknowledgements, rejections, trade confirmations and market-data feeds."
"How many clients and orders?"
"Zerodha had about 67.9 lakh active clients on NSE in September 2026, and took nearly 34.5 million orders on Lok Sabha election results day in 2024. Design for 7 million active clients, 15 million orders on a busy normal day and 40 million on a spike day."
"How much of that lands at the open?"
"The sharpest part. Zerodha's 2020 traffic graph showed peak activity 'as always' at 9:15, and after-market orders placed overnight are waiting to go out too."
"Must margin be checked before the order is sent?"
"Yes. Since September 2021 SEBI's peak-margin rules require full margin upfront. And SEBI's Stock Brokers Regulations, 2026 require records to be kept for eight years, up from five."
Functional Requirements#
- Orders: place, modify and cancel market, limit and stop-loss orders in equity, F&O and commodities; after-market orders queued overnight; one order per tap
- Risk: check and block margin before sending; release it on rejection or cancellation
- Updates and data: every order state change pushed within a second; live watchlist prices, market depth, charts
- Money: positions, holdings, P&L; add and withdraw funds; contract notes before the next trading day
Non-Functional Requirements#
Correctness
- Zero lost orders and zero duplicate orders sent to the exchange
- Margin check and block atomic per account: no two orders spend the same rupee
Latency (design targets, at our edge)
- Place order to "validated": p99 < 100 ms, also at 9:15
- Exchange acknowledgement to the client's screen: p99 < 1 s
- Exchange price update to bytes on the wire: p99 < 1.2 s
Availability and audit
- Order placement and cancellation are the last things to degrade; one line or data centre failing never takes the platform down
- Every order event append-only, kept 8 years; malfunctions that stop or slow trading for five minutes or more reported to the exchanges within two hours (SEBI's technical-glitch framework, as revised in January 2026)
Phase 2: Back of the Envelope Estimation (5 minutes)#
"The daily totals are not the problem. The open is. Let me size the tower."
Before the numbers: how tall is the 9:15 tower?
The normal market runs from 9:15 am to 3:30 pm. On a 15-million-order day, what is the average order rate? If 3% of the day's orders arrive in the first minute, what is the rate then?
One strong answer
- The session: 9:15 to 15:30 is 375 minutes = 22,500 seconds.
- The average: 15,000,000 ÷ 22,500 ≈ 670 orders a second.
- The first minute: 3% of 15 million = 450,000 in 60 s = 7,500 a second, about 11 times the average.
- The busiest second, assumed at twice the first minute's average: ~15,000 a second.
Sized for the average, the system fails in the day's most important minute. The good news: the tower is on the calendar.
Traffic Estimation#
| Normal busy day | Spike day | |
|---|---|---|
| Orders in the day | 15 million | 40 million |
| Average over 22,500 s | ~670/s | ~1,800/s |
| First minute (3% of the day) | 450,000 → 7,500/s | 1,200,000 → 20,000/s |
| Busiest second (2× the first minute) | ~15,000/s | ~40,000/s |
- Edge design capacity: 50,000 order requests a second, 1.25× the spike-day peak
- Reads at the open, if 1 million apps each make ~5 reads in the first 30 s: 5,000,000 ÷ 30 ≈ 167,000 a second
Conclusion: reads outnumber writes about 4 to 11 times (167,000 against 40,000 or 15,000), so they need their own path.
Market Data Estimation#
- 1.5 million WebSockets (rounded up from "more than a million" concurrent users) × 50 instruments (assumption) = 75 million subscriptions
- Design choice: one message per connection per second with the latest snapshot of each changed instrument; if 40% change each second (assumption), 75 M × 0.4 = 30 million packets a second, in line with Zerodha's "tens of millions"
- Kite's documented quote packet is 44 bytes, + 2-byte length = 46 bytes
- Per connection: 20 × 46 + 2-byte count = 922 bytes a second (~7.4 kbit/s)
- Total: 30 M × 46 B = 1.38 GB/s ≈ 11 Gbps, + ~60 bytes of framing and TCP/TLS per message × 1.5 M ≈ 0.7 Gbps → ~12 Gbps
- As JSON (~250 bytes a quote): 30 M × 250 B = 7.5 GB/s = 60 Gbps
- At 50,000 connections a server (assumption): 30 servers, doubled to 60 across two data centres
- Inbound (assumption): ~100,000 updates/s × ~150 B ≈ 120 Mbps, each fanned out ~300 times
Risk, Exchange and Storage Estimation#
- RMS state: 7 M accounts × ~2 KB = ~14 GB, in 16 in-memory partitions of ~0.9 GB
- Spike peak: 40,000 checks/s ÷ 16 = 2,500/s per partition; at ~20 µs each (assumption), 5% of one core
- Exchange messages at 1.3 per order (assumption): 40,000 × 1.3 = 52,000/s; at ~1,000 a session (illustrative; real limits are the exchange's), 52 sessions at 100%, so ~90 sessions at 60%
- Order events: ~5 per order × ~500 B → 37.5 GB on a normal day, 100 GB on a spike day; ~40 GB × 250 trading days ≈ 10 TB a year, ~80 TB over eight years
Key Metrics Summary#
| Metric | Value |
|---|---|
| Orders a day, normal / spike (design) | 15 million / 40 million |
| Average / first minute / peak second (normal day) | ~670/s / 7,500/s / ~15,000/s |
| Reads at the open | ~167,000/s |
| WebSockets / packets out | 1.5 million / ~30 million a second |
| Outbound market data, binary vs JSON | ~12 Gbps vs ~60 Gbps |
| Audit data over 8 years | ~80 TB |
Requirements and estimation
Q1/3 On a 15-million-order day with 3% of orders in the first minute, roughly how many orders a second arrive in that minute?
Answer & explanation
Answer: C
3% of 15 million is 450,000 orders in 60 seconds: 7,500 a second, about 11 times the day's average.
Q2/3 Why must the margin check happen before the order is sent?
Answer & explanation
Answer: B
Since September 2021 brokers must collect 100% of the required margin upfront, so an unfunded order must be stopped before it leaves.
Q3/3 Binary ticks need ~12 Gbps and JSON ~60 Gbps. Where does the saving come from?
Answer & explanation
Answer: D
A 44-byte quote packet is eleven 4-byte integers at known offsets. JSON repeats every field name and spells numbers as text.
Phase 3: High-Level Design (10 minutes)#
Sketch it first: one order, tap to fill
Trace Asha's order from the tap to the fill, and a watchlist price from the exchange to her screen. Which parts must be exact and durable, which can be served from memory, and which belong to someone else?
One strong answer
- Write path, exact and durable: gateway → order service (one order per idempotency key) → RMS (check and block margin, journalled) → admission queue → exchange gateway (sends once).
- Owned by the exchange: matching, acknowledgements, rejections and trades. For order state the exchange is the truth; the broker's job is to agree with it.
- Read path, from memory: orders, positions and margins served from per-user views in a hot cache, rebuilt from events.
- Market data, lossy by design: ticker servers keep only the latest snapshot per instrument. An old price is worthless once a newer one exists.
- Order updates, lossless: pushed over the same WebSocket but sequenced; on reconnect the app re-reads its order book.
"Orders are few and must be exact; prices are many and perishable; reads are most numerous and can be a moment stale. Each gets its own path."
System Architecture#
%% title: Trading platform architecture
flowchart TD
subgraph L1["Layer 1 · Clients"]
direction LR
APP@{ icon: "sd:phone", label: "Trading apps<br/>web, Android, iOS" }
API@{ icon: "sd:code", label: "API clients<br/>algos, partner apps" }
end
subgraph L2["Layer 2 · Edge"]
direction LR
GW["API gateway<br/>auth, per-client limits"]:::edge
WS@{ icon: "tech:websocket", label: "WebSocket edge<br/>long-lived connections" }
end
subgraph L3["Layer 3 · Read path (from memory)"]
direction LR
RD["Read API<br/>orders, positions, margins"]:::service
HOT@{ icon: "sd:cache", label: "Hot cache<br/>per-user views" }
TK["Ticker servers<br/>binary fan-out"]:::core
end
subgraph L4["Layer 4 · Write path (exact, audited)"]
direction LR
OMS["Order service<br/>state machine, idempotency"]:::core
RMS["RMS<br/>in-memory margin per account"]:::hot
ADM@{ icon: "sd:queue", label: "Admission queue<br/>priority lanes" }
end
subgraph L5["Layer 5 · Exchange side (two data centres)"]
direction LR
EGW["Exchange gateways<br/>sessions on leased lines"]:::service
FH["Feed handlers<br/>exchange market data"]:::service
EX@{ icon: "sd:building", label: "NSE · BSE · MCX" }
end
subgraph L6["Layer 6 · Records"]
direction LR
ODB@{ icon: "sd:log", label: "Order + audit store<br/>append-only, 8 years" }
LED@{ icon: "sd:database", label: "Funds ledger<br/>double entry" }
BUS@{ icon: "sd:stream", label: "Event bus<br/>order, fill, funds events" }
end
L1 ==> L2
L2 -->|"reads, ticks"| L3
L2 ==>|"place, modify, cancel"| L4
L3 -.-|"ticks from feed handlers"| L5
L4 ==> L5
L4 --> L6
class APP,API client
class WS edge
class HOT cache
class ADM,BUS queue
class EX external
class ODB,LED storeKEY INSIGHTS:
- The order service is the only writer of order state; order book, positions and P&L are projections of its events
- The RMS is in memory and partitioned by client: a check and block is one step, with no lock and no database round trip
- Ticker servers never touch the write path, so either side can struggle without stopping the other
Data Flow: One Order at 9:15:00#
%% title: One order at 9:15:00 sequenceDiagram autonumber participant A as Asha's app participant G as Gateway participant O as Order service participant R as RMS participant X as Exchange gateway participant E as Exchange Note over A: 9:15:00.120 · Buy 50 shares, limit, key k7 A->>G: Place order (key k7) G->>O: Admitted, bucket has a token O->>O: Insert once per (client, k7) · RECEIVED O->>R: Check and block margin R-->>O: OK, blocked and journalled O-->>A: 9:15:00.150 · Order 8812, VALIDATED O-)X: order.send event (outbox) X->>X: Mark SENT, then write the bytes X->>E: New order, broker reference 8812 E-->>X: Accepted, exchange order id X-)A: OPEN, pushed over the WebSocket E-->>X: Trade, 50 at the limit price X-)A: COMPLETE, position updated
MEANWHILE, IN PARALLEL:
- Asha's watchlist keeps ticking once a second from a ticker server that knows nothing about her order
- A million other apps refresh positions from the hot cache, never touching the order service
High-level design
Q1/3 Why do prices and order updates use different channels, even inside one WebSocket?
Answer & explanation
Answer: B
Prices are snapshots, so the next replaces the last. Order updates are events the client must see in order, so they are sequenced and replayable.
Q2/3 A million apps refresh their positions at 9:15:02. Where should those reads land?
Answer & explanation
Answer: C
Serving the read path from memory keeps a refresh storm away from order placement.
Q3/3 Who is the source of truth for the state of one order at the exchange?
Answer & explanation
Answer: D
The broker records what it sent and heard back; when the two disagree, it reconciles to the exchange.
Phase 4: Deep Dives (20 minutes)#
Deep Dive 1: The Order Lifecycle, Idempotency and Reconciliation (Week 2 Concepts: Idempotency, Week 5: Consistency)#
"An order has two lives: ours until we send it, and the exchange's after. The danger is at the handovers."
A double tap and a silent exchange
At 9:15:01 Asha's spinner hangs and she taps "Buy" again. Separately, an order is sent and the leased line drops before any acknowledgement. How do you make the double tap one order, and what do you do with the order nobody answered?
One strong answer
- One key per tap: the app creates an idempotency key on the tap and reuses it for every retry. A unique (client, key) constraint means a retry finds the existing order and gets its id and state back.
- Mark SENT before sending: a crash in between leaves "may have been sent", the safe belief.
- No answer is UNKNOWN, not FAILED: never resend it (it may be at the exchange) and never tell Asha it failed (she would place it again).
- Reconcile: ask the exchange, through a restored session or the other data centre, matching by your broker reference. Found: take the exchange's state. Not found after a full resync: NOT_PLACED, release the margin.
- Meanwhile, tell the truth: "Sent to the exchange, confirming. Don't place it again."
The Challenge:
- The response is lost on a mobile network and the app retries, or the user taps twice
- The order is sent and the acknowledgement never arrives
- A trade confirmation arrives before the "open" acknowledgement we expected
- A cancel races a fill
- Zerodha's 2019 postmortem shows the human side: orders stuck in "validation pending", and "customers placing more orders to make up for the hanging orders, increasing the pile up exponentially"
The Solution:
%% title: Order state machine stateDiagram-v2 [*] --> RECEIVED: inserted once per key RECEIVED --> VALIDATED: margin blocked RECEIVED --> REJECTED: RMS says no VALIDATED --> SENT: gateway marks, then sends VALIDATED --> REJECTED: queued past its deadline SENT --> OPEN: exchange accepts SENT --> REJECTED: exchange rejects SENT --> UNKNOWN: no answer in time UNKNOWN --> OPEN: reconciled, found UNKNOWN --> NOT_PLACED: reconciled, not found OPEN --> COMPLETE: filled OPEN --> CANCEL_PENDING: cancel sent OPEN --> CANCELLED: exchange cancels the rest CANCEL_PENDING --> CANCELLED: cancel confirmed CANCEL_PENDING --> COMPLETE: fill won the race UNKNOWN: UNKNOWN · may be at the exchange, never resend NOT_PLACED: NOT_PLACED · margin released REJECTED: REJECTED · reason shown, margin released class RECEIVED,VALIDATED,SENT,OPEN,CANCEL_PENDING core class COMPLETE ok class REJECTED,CANCELLED,NOT_PLACED bad class UNKNOWN queue
Kite's public API has the same shape: its documented statuses include PUT ORDER REQ RECEIVED, VALIDATION PENDING and OPEN PENDING before OPEN, COMPLETE, REJECTED or CANCELLED. Partial fills here are a growing filled quantity on an OPEN order.
# oms/orders.py
"""
Order placement and the order state machine (illustrative).
UNIQUE (client_id, idem_key) on the orders table makes placement idempotent.
Every state change, its audit row and its outbox events commit together.
"""
from dataclasses import dataclass
NEXT = {
"RECEIVED": {"VALIDATED", "REJECTED"},
"VALIDATED": {"SENT", "REJECTED"},
"SENT": {"OPEN", "REJECTED", "UNKNOWN"},
"UNKNOWN": {"OPEN", "REJECTED", "NOT_PLACED"},
"OPEN": {"COMPLETE", "CANCEL_PENDING", "CANCELLED"},
"CANCEL_PENDING": {"CANCELLED", "COMPLETE"},
}
class IllegalTransition(Exception):
"""Our view and the exchange's disagree: hand the order to reconciliation."""
@dataclass
class Order:
order_id: str
client_id: str
idem_key: str
qty: int
state: str = "RECEIVED"
filled: int = 0
exch_seq: int = 0 # last exchange message applied to this order
class OrderService:
def __init__(self, db, rms, outbox, audit, ids):
self.db, self.rms, self.outbox, self.audit, self.ids = db, rms, outbox, audit, ids
def place(self, client_id: str, idem_key: str, req: dict) -> Order:
with self.db.transaction() as tx:
existing = tx.order_by_key(client_id, idem_key)
if existing:
return existing # double tap or retry: same order, same answer
order = Order(self.ids.next(), client_id, idem_key, req["qty"])
tx.insert(order, req) # a racing duplicate fails here and re-reads
self.audit.append(tx, order, "RECEIVED")
verdict = self.rms.check_and_block(order) # idempotent on order_id; a sweeper re-drives
# orders a crash left in RECEIVED
with self.db.transaction() as tx:
if not verdict.ok:
return self._move(tx, order, "REJECTED", verdict.reason)
order = self._move(tx, order, "VALIDATED")
self.outbox.add(tx, "order.send", {"order_id": order.order_id})
return order
def mark_sent(self, order_id: str) -> bool:
"""The gateway calls this just BEFORE writing the order to the wire."""
with self.db.transaction() as tx:
order = tx.get_for_update(order_id)
if order.state != "VALIDATED":
return False # redelivered event, or shed
self._move(tx, order, "SENT")
return True
def on_exchange(self, order_id: str, seq: int, kind: str, qty: int = 0, reason: str = "") -> None:
with self.db.transaction() as tx:
order = tx.get_for_update(order_id)
if seq <= order.exch_seq:
return # duplicate or replayed exchange message
order.exch_seq = seq
if kind == "FILL":
if order.state in ("SENT", "UNKNOWN"):
order = self._move(tx, order, "OPEN") # a trade implies it was accepted
order.filled += qty
self.outbox.add(tx, "rms.fill", {"order_id": order_id, "qty": qty, "seq": seq})
if order.filled == order.qty:
order = self._move(tx, order, "COMPLETE")
else:
to = {"ACK": "OPEN", "REJECT": "REJECTED", "CANCELLED": "CANCELLED",
"NOT_FOUND": "NOT_PLACED"}[kind]
order = self._move(tx, order, to, reason)
if to != "OPEN":
self.outbox.add(tx, "rms.release", {"order_id": order_id, "filled": order.filled})
tx.save(order)
def _move(self, tx, order: Order, to: str, reason: str = "") -> Order:
if to not in NEXT.get(order.state, set()):
raise IllegalTransition(f"{order.order_id}: {order.state} -> {to}")
order.state = to
tx.save(order)
self.audit.append(tx, order, to, reason=reason)
self.outbox.add(tx, "order.update", {"order_id": order.order_id, "state": to, "filled": order.filled})
return orderA few details carry the correctness:
- The unique constraint is the idempotency: two requests with one key, even on two servers, create one row.
- SENT is written before the bytes leave: "send, then record" can lose the fact that an order went out; "record, then send" at worst makes an UNKNOWN.
- Exchange messages apply once, in order (last sequence per order), and RMS effects travel as outbox events, so margin never moves without its state change.
Reconciliation runs at three speeds: seconds, for every UNKNOWN order; minutes, a rolling comparison of open orders and trades with the exchange's records; and end of day, against the exchange's trade files before contract notes go out.
Deep Dive 2: Pre-Trade Risk at Open-Bell Scale (Week 1 Concepts: Partitioning and Hot Keys, Week 5: Consistency)#
"The RMS answers one question per order, up to 40,000 times a second at 9:15: can this client afford it right now? Two orders must never spend the same rupee."
Two orders, one rupee
A client has ₹1,00,000 available. At 9:15:00.004 two orders arrive from two devices, each needing ₹70,000, landing on two different order-service servers. At 9:14:50 the client also added ₹50,000 by UPI. Which orders go through, and how do you decide consistently without a database transaction per order?
One strong answer
- One owner per account: hash the client id to one RMS partition, whose single thread applies every change to the account in order. Two orders on two servers become two messages in one queue.
- Check and block in one step: without the pay-in, the first blocks ₹70,000 and the second sees ₹30,000 and is rejected: "required ₹70,000, available ₹30,000".
- Durable before answering: each block is appended to the partition's journal (fsync batched about every millisecond) before "OK". After a crash: morning snapshot plus journal replay.
- Money counts once the ledger has it: a posted pay-in reaches the partition as an event. If the ₹50,000 was posted at 9:14:50, available is ₹1,50,000 and both pass. If posting lags, the second is rejected with a clear reason and the client retries a moment later. The RMS never counts money the ledger hasn't recorded.
- A database row per account, locked per order: 40,000 lock-read-update-commit round trips a second, and busy API accounts queue behind their own row locks
- A cache with read-then-write: two orders both read "₹1,00,000 available" and both block ₹70,000
- Checking after sending: fast, and against the upfront-margin rule
The Solution:
%% title: Margin check and the funds ledger
flowchart TD
ORD(["Order from the order service"]):::client
ROUTE["Route by hash of client id<br/>to one of 16 partitions"]:::service
subgraph P["RMS partition 7 · one thread, in memory"]
direction TB
CHK{"required ≤ available?"}:::core
BLK["Block margin<br/>blocked += required"]:::hot
REJ(["Reject with both numbers<br/>required vs available"]):::bad
end
JRN@{ icon: "sd:log", label: "Journal<br/>append before answering" }
OK(["Validated, on to admission"]):::ok
LED@{ icon: "sd:database", label: "Funds ledger<br/>pay-ins, payouts, settlement" }
BUS@{ icon: "sd:stream", label: "Funds and fill events" }
ORD --> ROUTE --> CHK
CHK -->|"yes"| BLK --> JRN --> OK
CHK -->|"no"| REJ
LED -.->|"pay-in posted"| BUS -.->|"available rises"| P
class JRN,LED store
class BUS queue- available = cash + collateral − blocked; the exchanges' margin parameters are in memory before the open, so no check makes a remote call.
- Hot accounts are not hot partitions: even an API account at Kite Connect's documented 10 orders a second is a trickle beside ~2,500 checks a second per partition.
# rms/partition.py
"""
One RMS partition: owns the margin state of the accounts hashed to it.
Single-threaded, so an account's check-and-block has nothing in between.
Each change is journalled (fsync batched ~1 ms) before the caller hears
the answer; restart = pre-open snapshot + journal replay.
"""
from dataclasses import dataclass
from typing import Dict
@dataclass
class Account:
cash: int # paise: opening ledger balance plus today's postings
collateral: int # paise: haircut value of pledged holdings
blocked: int = 0 # paise: held by open orders and positions
ledger_seq: int = 0 # last funds-ledger posting applied
@property
def available(self) -> int:
return self.cash + self.collateral - self.blocked
@dataclass
class Verdict:
ok: bool
reason: str = ""
required: int = 0
available: int = 0
class RmsPartition:
def __init__(self, journal, margin_calc, accounts: Dict[str, Account]):
self.journal, self.calc, self.accounts = journal, margin_calc, accounts
self.held: Dict[str, int] = {} # order_id -> margin blocked for it
def check_and_block(self, order) -> Verdict:
if order.order_id in self.held: # re-driven after a crash: same answer
return Verdict(True, required=self.held[order.order_id])
acct = self.accounts.get(order.client_id)
if acct is None:
return Verdict(False, "ACCOUNT_NOT_LOADED")
need = self.calc.required(order, acct) # full value for a delivery buy, exchange margin for F&O
if need > acct.available:
return Verdict(False, "INSUFFICIENT_MARGIN", need, acct.available)
acct.blocked += need
self.held[order.order_id] = need
self.journal.append(("BLOCK", order.client_id, order.order_id, need))
return Verdict(True, required=need, available=acct.available)
def release(self, client_id: str, order_id: str, still_used: int = 0) -> None:
"""Rejected, cancelled or not placed. Safe to call twice."""
held = self.held.pop(order_id, None)
if held is None:
return
self.accounts[client_id].blocked -= held - still_used # fills keep their position margin
self.journal.append(("RELEASE", client_id, order_id, held - still_used))
def on_ledger(self, client_id: str, seq: int, amount: int) -> None:
"""A pay-in or end-of-day settlement posted to the funds ledger."""
acct = self.accounts[client_id]
if seq <= acct.ledger_seq:
return # replayed event
acct.cash += amount
acct.ledger_seq = seq
self.journal.append(("LEDGER", client_id, seq, amount))"Withdrawals go through the same partition thread as orders, so money can't leave while an order is using it. The funds ledger stays the source of truth for money; the RMS is a fast, exact projection of it plus today's orders, and at the end of the day the two must agree to the paisa."
Orders and margin
Q1/4 The app retries "place order" after a timeout with the same idempotency key; the first attempt actually succeeded. What happens?
Answer & explanation
Answer: B
The key belongs to the tap, so the retry can only read the order the first attempt created.
Q2/4 An order was sent and the line dropped before any acknowledgement. Which next step is safe?
Answer & explanation
Answer: D
It may already be at the exchange. Resending risks a duplicate; calling it failed invites one.
Q3/4 Why does each RMS partition apply an account's changes on one thread?
Answer & explanation
Answer: C
With one owner per account, "check, then block" has nothing in between.
Q4/4 A ₹50,000 UPI pay-in hasn't been posted to the ledger when a large order arrives. What does the RMS do?
Answer & explanation
Answer: B
The RMS counts only what the ledger has recorded; the client can retry a moment later.
Deep Dive 3: Market Data Fan-Out Over WebSockets (Week 3 Concepts: Backpressure, Week 8: Streaming)#
"The ticker is the most bandwidth-hungry part and the most forgiving: always show the latest price, and never let one slow phone hurt anyone else."
Thirty million packets a second
1.5 million apps each watch ~50 instruments; at the open ~40% change every second. One user is on a weak signal in a lift and reads nothing for 20 seconds. How do you keep everyone else live, keep bandwidth sane, and bring the user in the lift up to date afterwards?
One strong answer
- Binary, not JSON: fixed-layout packets (Kite's documented quote packet is 44 bytes, prices in paise) keep the stream near 12 Gbps instead of 60.
- Coalesce: keep only the latest snapshot per instrument; ten ticks in a second become one packet, because a snapshot carries full state, not a delta.
- Encode once, copy many: each update is encoded once and the same bytes go to every subscriber.
- Bound every socket: if a send buffer is over a limit, skip that connection this flush and remember what it missed; when it drains, it gets the latest values. Behind for ~10 s: close it, and the app reconnects and resubscribes.
- Scale by connections: every ticker server takes the whole (~120 Mbps) inbound feed and serves ~50,000 sockets; nothing is shared between servers.
- A text format five times bigger than needed, times a million screens
- Forwarding every exchange update to every subscriber
- Slow readers whose unsent bytes pile up in server memory
- A server restart that makes 50,000 apps reconnect and ask for snapshots in the same second
- Mixing order updates into the lossy stream, so a dropped packet is a missing "filled"
The Solution:
Kite Connect's public WebSocket API shows a practical format for this:
- Subscribe with a small JSON message (
{"a": "subscribe", "v": [408065]}) and pick a mode per instrument: ltp (8 bytes), quote (44 bytes) or full (184 bytes, with five levels of bids and offers) - Each binary message starts with a 2-byte packet count and each packet with a 2-byte length; prices are integers in paise; up to 3,000 instruments per connection; a 1-byte heartbeat keeps idle connections alive
- Order updates arrive on the same socket as JSON text messages, separate from the binary ticks
%% title: Market data fan-out
flowchart TD
EX@{ icon: "sd:building", label: "Exchange feeds<br/>~100,000 updates/s at the open" }
FH["Feed handlers, a pair per feed<br/>decode, normalise, check sequence"]:::service
BUS@{ icon: "sd:stream", label: "Internal tick bus" }
subgraph TS["One ticker server of ~60"]
direction TB
LAST@{ icon: "sd:cache", label: "Latest packet per instrument<br/>encoded once" }
FL["Flush every second<br/>changed and subscribed"]:::core
BP{"Socket buffer<br/>under 64 KB?"}:::service
end
SEND(["One binary message<br/>~20 packets, ~922 bytes"]):::ok
SKIP(["Skip this flush<br/>remember what it missed"]):::bad
EX --> FH --> BUS --> LAST --> FL --> BP
BP -->|"yes"| SEND
BP -->|"no"| SKIP
class EX external
class BUS queue
class LAST cacheTen ticks of instrument 408065 in one second overwrite one latest-packet slot; once a second the flush encodes each changed 44-byte packet once and sends the same bytes to every subscriber, about 20 packets and 922 bytes per phone. A phone that stops reading passes 64 KB unsent, so flushes skip it and remember what it missed; after about 10 s behind it is closed (4008, slow consumer), and it reconnects, resubscribes and gets the latest values.
# ticker/fanout.py
"""
One ticker server's fan-out loop (illustrative, asyncio). Ticks are
snapshots, which is what makes coalescing and skipping safe. Order
updates do NOT go through here; they use a sequenced channel.
"""
import asyncio
import struct
from typing import Dict, Set
FLUSH_S = 1.0 # one message per connection per second (tunable)
MAX_BUFFER = 64 * 1024 # bytes queued on a socket before we call it slow
SLOW_LIMIT_S = 10.0 # behind this long: close; the app reconnects
def quote_packet(token: int, q) -> bytes:
"""44 bytes in the layout Kite documents (eleven big-endian int32s,
prices in paise), preceded by a 2-byte length."""
body = struct.pack(">11i", token, q.ltp, q.last_qty, q.avg_price, q.volume,
q.buy_qty, q.sell_qty, q.open, q.high, q.low, q.close)
return struct.pack(">H", len(body)) + body
class Conn:
def __init__(self, ws):
self.ws = ws
self.missed: Set[int] = set() # instruments whose latest it hasn't seen
self.behind_since = None
class Ticker:
def __init__(self):
self.latest: Dict[int, bytes] = {} # token -> latest encoded packet
self.changed: Set[int] = set() # tokens updated since the last flush
self.subs: Dict[int, Set[Conn]] = {}
self.lagging: Set[Conn] = set()
def on_tick(self, token: int, quote) -> None:
self.latest[token] = quote_packet(token, quote) # encode once, share the bytes
self.changed.add(token) # newer overwrites older: coalescing
async def flush_loop(self) -> None:
loop = asyncio.get_running_loop()
while True:
await asyncio.sleep(FLUSH_S)
changed, self.changed = self.changed, set()
per_conn: Dict[Conn, Set[int]] = {c: set() for c in self.lagging}
for t in changed:
for c in self.subs.get(t, ()):
per_conn.setdefault(c, set()).add(t)
now = loop.time()
for c, toks in per_conn.items():
self._offer(c, toks | c.missed, now)
def _offer(self, c: Conn, toks: Set[int], now: float) -> None:
if not toks:
return
if c.ws.buffered_bytes() > MAX_BUFFER: # the phone isn't reading
c.missed = toks # newest values go out once it drains
self.lagging.add(c)
c.behind_since = c.behind_since or now
if now - c.behind_since > SLOW_LIMIT_S:
self.drop(c)
return
c.missed, c.behind_since = set(), None
self.lagging.discard(c)
pkts = [self.latest[t] for t in toks]
c.ws.send_nowait(struct.pack(">H", len(pkts)) + b"".join(pkts))
def drop(self, c: Conn) -> None:
for subs in self.subs.values():
subs.discard(c)
self.lagging.discard(c)
c.ws.close(code=4008, reason="slow consumer") # the app reconnects and resubscribesNew subscribers are simply added to missed for their instruments, so their snapshot goes out in the next flush.
Scaling it out:
- Connections are the unit of scale: each server holds ~50,000 sockets and the whole inbound feed, so doubling connections means doubling servers.
- Reconnect storms are spread by jittered exponential backoff in the app, and snapshots wait for the next flush.
- Order updates take another road: sequenced per user; on reconnect the app fetches its order book and applies only newer updates.
Market data
Q1/3 A stock ticks ten times in one second. How many packets for it does a subscriber receive that second?
Answer & explanation
Answer: A
Only the latest snapshot is kept and sent once per flush; nothing useful is lost.
Q2/3 A phone stops reading its socket for 20 seconds. What should the ticker server do?
Answer & explanation
Answer: B
One slow reader must not grow server memory or delay others; skipping is safe for snapshots.
Q3/3 Why must order updates not go through the coalescing path?
Answer & explanation
Answer: D
Ticks are snapshots; order updates are events that need ordering and replay.
Deep Dive 4: Surviving the Open Spike (Week 1 Concepts: Rate Limiting, Week 3: Backpressure, Week 10: Capacity Planning)#
"The 9:15 tower is the friendliest spike there is: we know to the second when it comes. Treat it as a scheduled event, not a surprise."
Planning for 9:15:00
It's 8:30 am. Orders will arrive at ~7,500 a second for a minute from 9:15:00, peaking near 15,000, and could be three times that on a big day. What do you do before 9:15, and what protects the order path at 9:15?
One strong answer
- Pre-warm, don't autoscale: the spike lasts a minute. By 8:45, scale on a schedule, load account snapshots and margin parameters into the RMS, warm caches, log sessions in.
- Do early what can be done early: after-market orders are validated before 9:15, so at the open they only need sending.
- Separate read and write capacity, so a refresh storm can't take a core from order placement.
- Admission control: per-client buckets, a global in-flight cap, priority lanes for cancels and exits, and a queue deadline after which a new order is rejected "not sent" rather than sent late.
- Degrade in a fixed order: charts first; never placement or cancellation.
- Autoscaling notices at 9:16 and adds capacity at 9:20, after the tower has passed
- Every app refreshes everything at once, on the same threads as order placement
- An unbounded queue sends orders seconds after the price the client saw, with cancels stuck behind new orders
- Slow answers trigger retries, and retries are more load
The Solution:
%% title: Admission at the open bell
flowchart TD
REQ(["Order request"]):::client
AUTH["Gateway<br/>session valid, request well formed"]:::edge
KIND{"Kind?"}:::service
TB{"Client bucket<br/>10/s, burst 20"}:::service
RL(["Rate limited · not placed"]):::bad
OMS["Order service + RMS"]:::core
subgraph Q["Priority lanes to the exchange"]
direction LR
C1[["1 · Cancel"]]:::queue
C2[["2 · Exit"]]:::queue
C3[["3 · Modify"]]:::queue
C4[["4 · New"]]:::queue
end
CAP{"In flight below<br/>sessions × 200?"}:::hot
SEND(["Sent to the exchange"]):::ok
STALE(["Past its queue deadline<br/>rejected · not sent"]):::bad
REQ --> AUTH --> KIND
KIND -->|"new, modify"| TB
KIND -->|"cancel, exit"| OMS
TB -->|"token"| OMS
TB -->|"empty"| RL
OMS --> Q --> CAP
CAP -->|"yes"| SEND
CAP -->|"waited too long"| STALE- Per-client bucket: 10 a second mirrors Kite Connect's documented API limit, and stops any retry loop taking more than its share. Cancels and exits always pass.
- Global in-flight cap: 90 sessions × 200 unacknowledged = at most 18,000 in flight; the rest wait in our queue, where we can still reorder or expire them.
- Queue deadlines (design targets): new 2 s, modify 3 s, exit 5 s, cancel 10 s. A new order past its deadline is rejected "not sent, nothing placed": the client decided at a price that has moved.
- Chart history from cache
- Market depth becomes quote mode, four times fewer bytes
- Tick flush stretches from 1 s to 2 s, half the packets
- Reports and settings say "busy, try in a few minutes"
- Never shed placement, modification, cancellation, exits or order updates
Deep Dive 5: When Things Break (Week 2 Concepts: Failure-First Design, Week 10: Incident Management)#
"The exchange has a problem, we have a partial one, or we disagree with the exchange. In each case the key output is a true sentence on the client's screen."
Prices frozen, one line flapping
At 10:08 the exchange's price feed stops updating, though no halt has been announced. Separately, one data centre's leased line keeps dropping for a few seconds and coming back. What do clients see, what happens to orders, and how do you route around the line?
One strong answer
- Detect staleness, not just disconnection: if active instruments stop updating while the session is up, show "Prices delayed since 10:08" instead of a frozen number that looks live.
- Be careful with market orders: warn, and rely on a market-protection limit (Kite documents a market protection parameter). If the exchange announces a halt, stop sending and say so.
- Treat flapping as down: a line that keeps recovering never looks "down" to failover logic. Judge sessions by recent success rate and acknowledgement latency, and move new orders to the other data centre when one degrades.
- Never move an in-flight order: orders already SENT on the bad line become UNKNOWN if unanswered and are reconciled through a healthy session. Moving them risks a duplicate.
- The exchange: on 24 February 2021 NSE stopped updating at about 10:08 and halted trading (F&O at 11:40, cash at 11:43) until 3:45 pm, after telecom-link failures made its clearing corporation's online risk-management system unavailable
- The broker's links: in May 2018 more than 3.5 lakh clients kept retrying logins at the open while Zerodha's leased lines "flapped": lines that "do not completely fail and hence do not automatically switch to the backup lines"
- A component: in February 2019 OMS connectivity problems at the open left several lakh orders in "validation pending"
- Part of the app, not the order path: on 3 February 2026, 9:15 to 9:42, margins, positions and UPI-intent fund additions were intermittent; Zerodha's disclosure said order placement, modification and cancellation were not affected, and listed alternatives (the Funds page, Kite Backup on WhatsApp, other payment methods)
The Solution:
%% title: An order lost in a session drop
sequenceDiagram
autonumber
participant O as Order service
participant X as Gateway in DC-A
participant E as Exchange
participant R as Reconciler
participant A as App
O->>X: Send order 8812
X->>E: New order, broker reference 8812
Note over X,E: Line drops before the acknowledgement
X-->>O: No answer in 2 s · UNKNOWN
O-)A: Sent to the exchange, confirming. Don't place it again
R->>E: Order status for reference 8812, via DC-B
alt found at the exchange
E-->>R: Open, exchange order id
R->>O: UNKNOWN to OPEN
else not found after full resync
E-->>R: No such order
R->>O: UNKNOWN to NOT_PLACED, release margin
end
O-)A: Final state, in plain wordsWhen we and the exchange disagree, the exchange wins: an unexpected fill is applied, an order the exchange cancelled (say the rest of an immediate-or-cancel order) moves to CANCELLED and releases margin, and a trade with no matching order pages a human. Each correction is an audit event too.
What the client sees matters as much as the state machine. Each state maps to one honest sentence:
| Internal state | What the screen says | What the client should do |
|---|---|---|
| RECEIVED / VALIDATED | Placing your order | Wait |
| REJECTED by RMS | Not placed: needs ₹70,000, available ₹30,000 | Add funds or change the order |
| REJECTED, queue deadline | Not sent, the system was busy. Nothing was placed | Try again if you still want it |
| SENT / UNKNOWN | Sent to the exchange, confirming. Don't place it again | Wait; it will update |
| OPEN | Open at the exchange | Modify or cancel if needed |
| NOT_PLACED | The exchange never received this order. Margin released | Place again if you still want it |
| REJECTED by exchange | Rejected by the exchange: the exchange's reason | Fix and retry |
"'Something went wrong' on an order that is open at the exchange is how one order becomes three."
The open spike and failures
Q1/4 Why is open-bell capacity scaled up on a schedule rather than left to autoscaling?
Answer & explanation
Answer: C
An autoscaler reacts after the load arrives and takes minutes; the tower is over by then.
Q2/4 Under heavy load, which request should reach the exchange first?
Answer & explanation
Answer: B
A delayed cancel can become an unwanted fill, so cancels and exits overtake new orders.
Q3/4 A new order has waited three seconds in the admission queue. Why reject it rather than send it late?
Answer & explanation
Answer: D
"Not sent, nothing placed" lets the client decide again instead of trading on a stale decision.
Q4/4 One data centre's leased line keeps dropping and reconnecting. What should the gateway do?
Answer & explanation
Answer: C
A flapping line never looks "down"; judge it by success and latency, and reconcile, never resend.
Phase 5: Scaling and Edge Cases (5 minutes)#
"Walk me through one trading day, and then the cases that bite."
A Trading Day#
- Evening before · Settle and send
- Reconcile the day's trades with the exchanges' files; post settlement to the funds ledger
- Contract notes to everyone who traded, before the next trading day (Zerodha generates 1.5 million+ in about 25 minutes, per its 2024 tech blog)
- 8:30-9:00 AM · Load and scale
- RMS snapshot and margin parameters loaded; open-bell capacity in place; sessions logged in
- 9:00-9:15 AM · Pre-open call auction (equities)
- Order entry until about 9:10 (as of September 2026; closing at a random moment in its last two minutes), price discovery and matching until 9:12, a buffer until 9:15
- 9:15 AM · The open
- The day's sharpest load
- 9:15 AM-3:30 PM · Continuous trading
- Scheduled spikes (a budget speech) and unscheduled ones (results, global news); reconciliation running throughout
Edge Cases#
Edge case 1: A fill beats a cancel
Problem: The order fills a millisecond before the exchange processes the client's cancel.
Solution: CANCEL_PENDING → COMPLETE is legal; the screen says "Filled before your cancel reached the exchange", not "cancel failed".
Edge case 2: A rejection storm at the open
Problem: Thousands of limit orders are rejected at once, for example for prices outside the day's band.
Solution: Show the exchange's reason in plain words, release margin at once, and pre-check known price bands so out-of-band orders never leave.
Edge case 3: An RMS partition crashes at 9:15
Problem: One of 16 partitions fails at the busiest moment.
Solution: A standby following the journal takes over in seconds; only that partition's clients wait.
Scaling and edge cases
Q1/3 A client cancels and the exchange reports that the order filled first. How is it recorded?
Answer & explanation
Answer: B
The exchange's trade is a fact, and the honest message prevents a confused re-cancel.
Q2/3 Why are after-market orders validated against the RMS before 9:15?
Answer & explanation
Answer: A
Anything that can happen before the tower should.
Q3/3 One RMS partition crashes at 9:15. What is the blast radius?
Answer & explanation
Answer: C
Partitions are independent; a standby replays the journal and takes over.
Phase 6: Monitoring and Operations#
"The minute that matters is the first one, so the dashboard is about the open: how fast orders are validated and acknowledged, how deep the queue is, and whether prices are live."
Monitoring Dashboard#
Orders
| Metric | Now |
|---|---|
| Order requests, last 10 s | 61,400 (6,140/s) |
| Place → validated p50 / p99 | 18 / 74 ms |
| Exchange acknowledgement p50 / p99 | 6 / 31 ms |
| In flight / cap | 74 / 18,000 |
| Admission queue depth | 0 |
| Orders in UNKNOWN | 3 |
Risk and market data
- RMS checks 6,150/s, busiest partition 460/s; margin rejections 4.1% of new orders
- Sockets 1.41 million on 60 ticker servers; worst feed lag 120 ms; stale feeds 0; slow-consumer closes last minute 2,310
Alerts (today)
- 09:15:04 (amber) — Session NSE-A-17 acknowledgement p99 at 400 ms, new orders shifted to other sessions
- 09:15:09 (green) — NSE-A-17 under 40 ms for 60 s, restored
- 09:15:21 (amber) — Read path at 92% of capacity, chart history switched to cached mode
SLOs for the Platform#
SLO 1: Order integrity
- Target: 0 lost orders and 0 duplicate exchange orders; error budget: none, any reconciliation mismatch is an incident
SLO 2: Placement latency
- Target: place → validated p99 < 100 ms in every minute of the session, including 9:15-9:16 (design target)
SLO 3: Unknowns resolved
- Target: 99.9% of UNKNOWN orders resolved within 30 s, all within 5 minutes
SLO 4: Live, honest prices
- Target: exchange update to wire p99 < 1.2 s; stale feeds flagged to clients within 5 s
Reading the dashboard
Q1/3 The placement-latency SLO is measured per minute, not per day. Why?
Answer & explanation
Answer: C
A slow first minute barely moves the daily figure, yet that is when most clients trade.
Q2/3 At 6,140 orders a second, each in flight for about 12 ms on average, only ~74 are in flight. Which law gives that?
Answer & explanation
Answer: A
6,140 × 0.012 s ≈ 74. The same law sets the throughput ceiling when the exchange slows down.
Q3/3 Why is "orders in UNKNOWN" on the front page of the dashboard?
Answer & explanation
Answer: C
Lingering UNKNOWNs become duplicate orders and support calls; a rising count means an unhealthy path.
Final Challenge: Budget Day, and the Exchange Is Slow#
The interviewer scrolls to a graph with two towers. "One more."
"It's Union Budget day. On Budget day 2024 NSE received more than 2,044 crore orders, then a world record for any exchange. Suppose at 11:40 the finance minister announces a change to capital-gains tax. Your order rate jumps to 30,000 a second for a minute, and the exchange, under its own load, slows its acknowledgements from about 5 ms to about 800 ms. Spinners spin; clients retry. Keep orders correct, keep the ticker live, and decide what to shed."
- Demand: 30,000 new orders a second for about a minute, plus cancels and exits
- Exchange acknowledgement: ~5 ms → ~800 ms
- Our cap: 90 sessions × 200 unacknowledged = 18,000 in flight
- About 30% of clients retry once if the spinner lasts over three seconds
- The exchange's price feed is heavier but still flowing
What do you protect, and what do you shed?
Work out how many orders a second can reach the exchange at 800 ms per acknowledgement, what happens to the queue, what retries do to it, and what the platform should do.
One strong answer
- The ceiling (Little's law): rate = in flight ÷ latency = 18,000 ÷ 0.8 s = 22,500 orders a second. At 5 ms the same cap allowed 3.6 million a second; the exchange's latency, not our servers, is now the limit.
- The queue: 30,000 against 22,500 grows the backlog by 7,500 a second, 75,000 after 10 s, so a new order would wait 75,000 ÷ 22,500 ≈ 3.3 s, past its 2 s deadline. Some clients will see "not sent, the system was busy", which is correct.
- Retries: if they create new orders, demand becomes 39,000 a second, the backlog grows 16,500 a second, and each retry that gets through is a duplicate position. With one key per tap, a retry just reads the existing order: a cheap read at the gateway, nothing added to the exchange queue.
- Protect: cancels and exits in the top lanes; order updates; UNKNOWN reconciliation; the RMS.
- Shed on the read side: chart history to cache, depth to quote, reports off. The ticker has its own servers and a flowing feed, so it stays live.
- Tell the truth: "The exchange is responding slowly. Your order is sent and confirming. Don't place it again." Five minutes or more of stopped or slowed trading is a reportable technical glitch.
Final challenge · Quest check
Q1/4 With at most 18,000 orders in flight and acknowledgements taking 0.8 s, roughly how many orders a second can reach the exchange?
Answer & explanation
Answer: B
Little's law: throughput = in flight ÷ latency = 18,000 ÷ 0.8 ≈ 22,500 a second, so demand of 30,000 a second builds a queue.
Q2/4 A spinner has run for four seconds and the app retries with the same idempotency key. What does that add to the exchange queue?
Answer & explanation
Answer: C
The key is per tap, so retries become cheap reads, not duplicate positions.
Q3/4 The queue is growing. Which requests keep their priority?
Answer & explanation
Answer: D
On a fast day, letting clients get out matters most; charts and depth are shed first.
Q4/4 The exchange is slow to acknowledge but its price feed is flowing. What happens to the ticker?
Answer & explanation
Answer: A
The ticker never touches the write path, and clients deciding whether to cancel need live prices.
Interview Conclusion#
"Why not just retry the exchange send when an acknowledgement times out?"
"Because the order may already be resting at the exchange. A timeout means 'I don't know', and the only safe move from 'I don't know' is to ask, not to act again."
"Where would you spend your next engineering quarter?"
"On an open-bell drill: replay a recorded big-day 9:15 against staging every week, with the exchange simulator slowed and one line flapping, and measure placement latency, UNKNOWN counts and message accuracy. The tower comes every morning; production should never be the first place the system meets it."
"The single most important decision?"
"Separating the traffic by what it can afford to lose. Orders lose nothing and get the exact, journalled, idempotent path. Prices can drop everything but the latest. Reads can be a moment stale. Once those are apart, 9:15 is a capacity problem, not a correctness problem."
Summary: Concepts Applied from 10-Week Course#
- Week 1: Data at scale
- Partitioning — RMS state hashed by client across in-memory partitions
- Rate limiting — per-client token buckets on new orders and modifications
- Week 2: Failure-first design
- Idempotency — one key per tap, enforced by a unique constraint
- Timeouts — an unanswered order is UNKNOWN, reconciled rather than resent
- Week 3: Messaging and async processing
- Transactional outbox — order state, RMS effects and client updates commit together
- Backpressure — bounded sockets, skipped flushes, a capped in-flight window
- Audit log — append-only order events kept for eight years
- Week 5: Consistency and coordination
- Single writer per account; the exchange as source of truth for order state
- Week 8: Streaming
- Binary, coalesced market-data fan-out beside a sequenced order-update stream
- Week 10: Production readiness
- Capacity planning for a scheduled spike, per-minute SLOs, a fixed degradation ladder
Why Zerodha Kite Matters#
The spike is on the calendar
- Most systems fear unpredictable load. A broker knows its worst second: 9:15:00 every trading day, bigger on budget, election and results days
Money and time at once
- Each order is exact money and a decision about a moving price; correctness and latency can't be traded off
Someone else owns the truth
- The exchange matches and confirms. The broker's craft is agreeing with it, and telling the client honestly when it doesn't know yet
"At 9:15 the market waits for no one. The platform's job is to make every tap exactly one order, and to tell every client which state it is in."
Self-Assessment Checklist#
After studying this case study, you should be able to:
Architecture:
- Separate orders, market data and reads into paths with different loss tolerances
- Trace one order from tap to fill, naming who owns each state
Orders and risk:
- Make order placement idempotent with one key per tap
- Design an order state machine with an UNKNOWN state and reconciliation
- Partition margin state so each account has a single writer, consistent with a funds ledger
Market data:
- Size a WebSocket fan-out and explain what binary packets and coalescing save
- Handle slow consumers and reconnect storms
Operations:
- Plan a scheduled spike with pre-warming, admission control and priority lanes
- Use Little's law to find the throughput ceiling when the exchange slows
- Write the client-facing message for every order state
Sources#
Zerodha's own publications:
- Blog
Zerodha Tech — Scaling with common sense (Go, a hot Redis cache for everything shown on Kite, hundreds of thousands of ticker WebSocket connections), June 2020: https://zerodha.tech/blog/scaling-with-common-sense/
- Blog
Zerodha Tech — Scaling with common sense #2: Being future ready (1 million+ concurrent users, "tens of millions" of market-data packets a second, NATS, Kafka, 40 ms baseline), July 2021: https://zerodha.tech/blog/being-future-ready-with-common-sense/
- Blog
Zerodha Tech — 1.5+ million PDFs in 25 minutes (contract notes), February 2024: https://zerodha.tech/blog/1-5-million-pdfs-in-25-minutes/
- Blog
Z-Connect — Today's technical issues: postmortem (9:14-9:20 activity, OMS connectivity, "validation pending", orders piling up), February 2019: https://zerodha.com/z-connect/general/todays-technical-issues-postmortem
- Blog
Z-Connect — We were not down today (peak at 9:15, two OMS in two data centres, leased-line re-connections 9:15-9:18), May 2020: https://zerodha.com/z-connect/wp-json/wp/v2/posts/255009
- Read
Z-Connect — On conspiracy theories about Zerodha (several lakh logins at 9:15), September 2019: https://zerodha.com/z-connect/general/on-conspiracy-theories-about-zerodha
- Read
Zerodha disclosure — Issue with display of margins, funds addition and positions, 3 February 2026: https://zerodha.com/marketintel/disclosures/441367/issue-with-display-of-margins-on-the-order-window-funds-addition-and-display-of-positions
Kite Connect documentation:
- Blog
WebSocket streaming (ltp/quote/full modes, 8/44/184-byte packets, message layout, prices in paise, 3,000 instruments per connection, order postbacks): https://kite.trade/docs/connect/v3/websocket/
- Docs
Orders (order statuses, varieties, market protection): https://kite.trade/docs/connect/v3/orders/
- Docs
Exceptions and rate limits (10 orders a second per user): https://kite.trade/docs/connect/v3/exceptions/
Press:
- Read
Business Today — Zerodha, Groww, Angel One, Upstox: active clients over eight years (68.93 lakh in March 2026), April 2026: https://www.businesstoday.in/markets/stocks/story/zerodha-groww-angel-one-upstox-how-active-clients-changed-in-past-8-years-full-table-526520-2026-04-20
- Read
Entrackr — Groww adds over 1 lakh active clients in September (Zerodha at 67.91 lakh NSE active clients, September 2026), October 2026: https://entrackr.com/news-analysis/groww-adds-over-1-lakh-active-clients-in-september-sahi-clocks-11-growth-12636259
- Read
IBTimes India — Activity on Kite on Lok Sabha election results day (34.5 million orders, 8 million+ logins), June 2024: https://www.ibtimes.co.in/over-rs-8k-cr-added-zerodhas-kite-app-lok-sabha-election-results-day-868853
- Read
Business Today — What is OmneNEST, Zerodha's backend tech, August 2023: https://www.businesstoday.in/latest/corporate/story/what-is-omnenest-zerodhas-backend-tech-that-drives-their-business-393124-2023-08-07
- Blog
Moneylife — Zerodha blames Metro digging in Mumbai for two-hour outage on 2 May 2018 (flapping leased lines): https://www.moneylife.in/article/zerodha-blames-metro-digging-in-mumbai-for-twohour-outage-on-2nd-may/53897.html
- Blog
Moneylife — NSE trading halt of 24 February 2021 (timeline and cause): https://www.moneylife.in/article/nse-trading-halt-rbi-says-failure-at-nse-clearing-led-to-outage-on-24th-february/63276.html
- Read
All India Radio News — NSE received 2,044 crore orders on Budget day, 23 July 2024: https://www.newsonair.gov.in/bse-md-ceo-sundararaman-ramamurthy-terms-union-budget-2024-25-as-forward-looking-equitable-and-middle-class-oriented
- Blog
Moneylife — Explainer on SEBI's peak margin norms from 1 September 2021: https://moneylife.in/article/explainer-how-sebis-new-peak-margin-norms-impact-traders-from-1st-september/64993.html
Exchange and regulator:
- Read
NSE — Equity market pre-open session (9:00-9:15 call auction): https://www.nseindia.com/static/products-services/equity-market-pre-open
- Read
Kotak Neo — NSE pre-open session gets new order rules from 7 September 2026 (order entry to 9:10 with random closure in the last two minutes, matching 9:10-9:12): https://www.kotakneo.com/news/trading/nse-pre-open-session-new-order-rules-7-september/
- Read
SEBI — Framework to address technical glitches in stock brokers' electronic trading systems, November 2022: https://www.sebi.gov.in/legal/circulars/nov-2022/framework-to-address-the-technical-glitches-in-stock-brokers-electronic-trading-systems_65466.html
- Read
SEBI — Review of the technical-glitch framework (brokers with more than 10,000 clients, reporting within two hours), January 2026: https://www.sebi.gov.in/legal/circulars/jan-2026/review-of-framework-to-address-the-technical-glitches-in-stock-brokers-electronic-trading-systems_99006.html
- Read
LiveLaw — SEBI notifies Stock Brokers Regulations, 2026 (records kept eight years), January 2026: https://www.livelaw.in/news-updates/sebi-notifies-stock-brokers-regulations-2026-notified-allows-stock-brokers-to-take-up-news-regulated-activities-518264
Design note: the architecture, code, thresholds, rates and dashboards here are an illustrative design for the problem, not a description of Zerodha's internal systems beyond what Zerodha has published. Nothing in it is investment advice.
Further Reading#
Related systems to study:
- Read
IRCTC Tatkal (Bonus Problem 11 in this series): another stampede at a known second
- Read
UPI (Bonus Problem 1): idempotency and "pending" states between parties
- Read
Twitter / X (Bonus Problem 10): fan-out at scale
Foundational concepts (this course):
- Read
Week 1, Day 3 — Rate Limiting at Scale
- Read
Week 2, Day 2 — Idempotency in Practice
- Read
Week 3, Day 3 — Backpressure and Flow Control
- Read
Week 3, Day 5 — Audit Log System
- Read
Week 10, Day 4 — Capacity Planning
- Read
Week 10, Day 5 — Incident Management
"The tower comes at 9:15 every morning. A good trading platform has met it a thousand times before it happens, and still tells every client the truth about their order."
- 0/7stages cleared
Go deeper · reading trail
all reading →Real write-ups from the teams that built these systems. Each opens on the publisher's site in a new tab.
Scaling with common sense
Kailash Nadh · 2020- What
- Zerodha's CTO on the principles behind running India's largest stock broker with a small team.
- Key idea
- Keep it simple, find the global bottleneck (usually the database) before micro-optimising, and trust a well-partitioned RDBMS before reaching for exotic tools.
- Why here
- The mindset behind the Kite design in this quest, from the people who run it.
The LMAX Architecture
Martin Fowler · 2011- What
- Martin Fowler's description of LMAX, a retail trading platform processing millions of orders a second on one thread.
- Key idea
- Business logic runs single-threaded in memory, fed by Disruptor ring buffers that handle input, journaling and replication around it.
- Why here
- The order path of this quest pushed to the extreme: sequence orders, keep state in memory and rebuild it from the journal.
1.5+ million PDFs in 25 minutes
Sarat Chandra, Karan Sharma · 2024- What
- How Zerodha generates, signs and emails over 1.5 million contract notes after market close.
- Key idea
- A chain of jobs (generate, sign, email) runs across workers through Tasqueue, Zerodha's lightweight Go job library, using EC2 and S3 for scratch storage.
- Why here
- The after-the-bell half of a trading day in this quest: a regulatory deadline turns into a distributed batch job.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.