Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.
How Do You Take a Payment Exactly Once When the Bank on the Other End Sometimes Never Replies?
It is 11:59:58 PM. An online store's festive sale starts at midnight. Forty thousand shoppers have full carts and their thumbs over "Pay". In two seconds, every one of them will press it.
Behind the button is a chain the shopper never sees: the store's server, the payment gateway, an acquiring bank or a UPI switch, a card network or NPCI, and the shopper's own bank, which has to ask for a PIN or an OTP. Any link can be slow. Some will not answer at all. And for every payment, the gateway must end up with one answer it can stand behind: the money moved, or it didn't.
If it guesses wrong, either a shopper is charged for an order the store never saw, or a store ships an order nobody paid for. At a billion transactions a quarter, "rarely" happens every minute.
- $180 billion+ — Razorpay's annualised payment volume, and 1 billion+ transactions a quarter (its CEO, Business Today, August 2026)
- 24.07 billion — UPI transactions in India in September 2026, worth ₹29.37 lakh crore, an average of 802 million a day (NPCI data)
- 400 TPS — the peak, during the IPL, of the first version of Razorpay's UPI switch; the second version was designed for 10,000 TPS (Razorpay engineering, September 2026)
- 70%+ — share of Razorpay's UPI volume that runs through that switch
- T+2 working days — Razorpay's default settlement cycle for domestic payments; instant settlement is optional (Razorpay docs)
- 3 days — an authorised payment not captured by then is refunded to the customer automatically (Razorpay docs)
- 24 hours, 5 seconds — webhooks are retried with exponential backoff for 24 hours, and each delivery must be answered within 5 seconds (Razorpay docs)
For context:
Razorpay received the RBI's final authorisation as a payment aggregator in December 2023
The RBI's Master Direction on payment aggregators (15 September 2025) requires a non-bank aggregator to keep merchants' funds in an escrow account with a scheduled commercial bank
Since 1 October 2022, merchants can't store customers' card numbers; card-on-file payments use network tokens
Everything in this design that is not in the list above is an illustrative design target, not Razorpay's internal architecture
This is the system we'll design today.
Razorpay handles $180 billion+ a year and over a billion payments a quarter; designed for 10,000 TPS, each payment fans out into ~15 writes and events, and ~13,500 payments are unknown at any moment because NPCI allows only three UPI status checks 90 s apart. Idempotent APIs feed a payment state machine, a router picks processors by success rate per slice, state changes commit with ledger entries and an outbox, and a reconciler resolves unknowns. The deep dives cover idempotency, routing, unknown outcomes, the ledger and T+2 settlement and webhooks, then a sale midnight, one slow issuer and the daily three-way match.
The Interview Begins#
You're interviewing at a payments company. The principal engineer draws a line of boxes across the whiteboard: Store → Gateway → Bank → Network → Bank. Then she draws a question mark over the last arrow.
"Every payments interview ends up at this question mark. A business integrates with us once and wants to accept every method India has. Design the gateway: taking the payment, routing it, knowing whether it worked, telling the merchant, and paying them. Then tell me what you do when the question mark doesn't answer."
Businesses create orders through an API and send customers to a checkout. Customers pay by card, UPI, net banking or wallet. The gateway sends each payment to one of several banks and processors, learns the result, informs the merchant, and settles the money to the merchant's bank account.
Requirements
- A customer is charged at most once per order, however many times anyone retries
- Every payment ends in exactly one final state: captured and settled, or failed or refunded
- Route each payment where it is most likely to succeed; survive any one bank or processor failing
- Tell the merchant about every state change, reliably, even when their server is down
- Pay merchants the right amount on the agreed schedule, from funds held in escrow
- Never store card numbers outside a secured vault; follow RBI and PCI-DSS rules
"I care about one word: exactly. Show me where 'exactly once' is real, and where you build it out of 'at least once' plus something clever."
Phase 1: Requirements Clarification (5 minutes)#
"Before boxes, I want to know which parts we control, and how slow the parts we don't control can be."
Your Questions#
"Do we hold the money, or does it go straight from the customer's bank to the merchant's?"
"As a payment aggregator, the money lands in an escrow account at a bank, held for the merchants, and we instruct payouts from it. The RBI's 2025 Master Direction requires exactly that for non-bank aggregators. Razorpay's default settlement is T+2 working days, with instant settlement as an option."
"Which methods, and which ones are asynchronous?"
"Cards, UPI, net banking, wallets. UPI is asynchronous by nature: you send a request, the customer approves it in their UPI app, and the result comes back later as a callback. Razorpay's engineers describe a message bus at the heart of their UPI switch for exactly that reason. Cards usually need an authentication step at the customer's bank, which can be slow."
"What's the volume?"
"Razorpay says it handles more than a billion transactions a quarter and more than $180 billion a year. The first version of its UPI switch peaked at 400 TPS during the IPL; the next was built for 10,000. Size it."
"When a bank doesn't answer, can we just ask it again?"
"Within limits. For UPI, NPCI's rules since August 2025 allow a transaction status check only three times, at least 90 seconds apart. So you can't poll your way out of uncertainty."
"What does the merchant integrate with?"
"An API to create orders and fetch payments, a checkout the customer sees, and webhooks to the merchant's server for every state change. Payments move through created, authorised and captured; an authorised payment that isn't captured within 3 days is refunded automatically."
"And card data?"
"Since October 2022, merchants can't store card numbers; saved cards use network tokens. The card data you do touch lives in a PCI-DSS vault, and the RBI's authentication rules from April 2026 require two factors, one of them dynamic, with issuers free to add risk-based checks."
Functional Requirements#
1. Merchant API
- Create an order (amount, currency, receipt); fetch orders, payments, refunds
- Every write API takes an idempotency key
- Capture an authorised payment (automatically or by API); refund fully or partly
2. Checkout
- Show methods by merchant configuration and device; tokenised saved cards
- Collect card details into the vault, or start UPI intent / QR / collect, or redirect to net banking
3. Processing
- Route each payment to a processor (acquiring bank, UPI switch, net-banking connection) by method, bank and live success rates
- Run authentication (card OTP or other factors, UPI PIN in the customer's app)
- Track every payment to a final state; resolve unknowns by status checks and settlement files
4. Money
- Double-entry ledger of every movement: captured, fees, taxes, refunds, chargebacks, settlements
- Settle to merchants on schedule (T+2 by default, instant on request) from escrow
5. Notifications
- Webhooks for every state change, signed, retried, with event ids for deduplication
- Dashboard and reports for merchants
Non-Functional Requirements#
Scale
- ~1 billion+ transactions a quarter; design for a 10,000 TPS peak (design target, the same order as the UPI switch's own target)
Correctness
- At most one capture per order; no payment left in a non-final state beyond its reconciliation window
- The ledger always balances; settlements equal captured minus fees, taxes and refunds
Latency
- Gateway's own work (create, route, persist) < 100 ms p99, excluding the bank and the customer
- Payment result to the merchant's server: webhook sent < 5 s after we learn it
Availability
- 99.99% for the payment path (design target); any one processor or bank can fail without taking the gateway down
- One merchant's spike can't starve other merchants
Security and compliance
- PCI-DSS card vault; no card numbers outside it; tokens for saved cards
- Funds in escrow; merchant due diligence before onboarding; audit trail for every money movement
Phase 2: Back of the Envelope Estimation (5 minutes)#
"The transaction rate is modest by internet standards. What makes payments heavy is the number of things that happen per payment, and the number of payments whose outcome we don't know yet."
Before the numbers: how many unknowns?
At the 10,000 TPS design peak, suppose 1% of UPI payments (say half of all traffic) haven't got a definite answer 30 seconds after the customer approved. NPCI allows a status check only three times, 90 seconds apart. How many payments are in "unknown" at any moment, and what does that tell you?
One strong answer
- New unknowns: 10,000 × 50% × 1% = 50 a second.
- How long they stay unknown: if the first status check is at 90 s and the last at 270 s, an unresolved payment can stay unknown for ~4.5 minutes, longer if only the settlement file settles it.
- How many at once: 50 × 270 s ≈ 13,500 payments in "unknown" at the peak, plus a tail waiting for the next day's files.
So "unknown" isn't an error path; it's a normal state with its own storage, timers and reconciliation. The design must keep the customer and the merchant informed while those 13,500 resolve, and must never let a retry turn one of them into two charges.
Traffic Estimation#
- Public baseline: 1 billion+ transactions a quarter = ~11 million a day, ~127 a second on average
- Peaks: sale midnights, festive evenings and match nights, assumed ~10× average for the busiest minutes: ~1,300 TPS
- Design target with headroom: 10,000 TPS
- Average transaction: $180 billion ÷ ~4 billion a year ≈ $45
- Internal work per payment (order, payment, routing decision, processor call, callback, capture, ~4 ledger entries, ~3 webhooks, notifications): ~15 writes or events
- At the design peak: 10,000 × 15 = ~150,000 internal operations a second
- Webhooks: ~3 per payment → ~30,000 a second at the design peak, each with up to 24 hours of retries if a merchant's server is down
Storage Estimation#
- Payment records: 11 million a day × ~2 KB (amount, method, token reference, processor references, state history, risk score) = ~22 GB a day
- Ledger entries: ~4 per payment × ~200 bytes = ~9 GB a day
- Webhook delivery log: ~3 per payment × ~1 KB = ~33 GB a day, kept hot for weeks
- Retention for money records (design assumption): 10 years → payments and ledger ≈ 31 GB × 365 × 10 ≈ ~113 TB
- Card vault: tokens and encrypted card data in a separate, small, heavily isolated store
Key Metrics Summary#
| Metric | Value |
|---|---|
| Average rate | ~127 TPS |
| Busiest minutes (assumed 10×) | ~1,300 TPS |
| Design peak | 10,000 TPS |
| Internal operations at the design peak | ~150,000/s |
| Webhooks at the design peak | ~30,000/s |
| Payments in "unknown" at the peak | ~13,500 |
| Payments + ledger, 10 years | ~113 TB |
Requirements and estimation
Q1/3 1 billion transactions a quarter is roughly how many a second on average?
Answer & explanation
Answer: B
1,000,000,000 ÷ 91 days ÷ 86,400 s ≈ 127 a second. The design peak of 10,000 TPS leaves room for sale midnights and growth.
Q2/3 Why does the estimate count ~15 internal operations per payment?
Answer & explanation
Answer: C
The capacity that matters is internal: at 10,000 TPS the gateway does ~150,000 writes and events a second. Sizing only for "payments" undersizes the system 15×.
Q3/3 About 13,500 payments sit in "unknown" at the design peak. What does that mean for the design?
Answer & explanation
Answer: C
NPCI allows only three status checks, 90 s apart, so uncertainty lasts minutes. Treating it as a first-class state is what keeps retries from becoming double charges.
Phase 3: High-Level Design (10 minutes)#
Sketch it first: one payment, from "Pay" to settlement
Trace one ₹1,999 card payment from the merchant creating an order to the money arriving in the merchant's bank. Which steps must be exactly once, which are at least once with deduplication, and which can run hours later?
One strong answer
- Exactly once (by construction): creating the order and the payment, keyed on idempotency keys, and the capture, which is a state transition on one row guarded by its current state.
- At least once, deduplicated: processor callbacks, status checks, webhooks to the merchant (with event ids), retries from the merchant's server. Each is safe to repeat because the receiving side checks state or ids.
- The payment path: checkout → vault (card to token) → router (pick a processor by live success rates) → processor and bank, with authentication → result → state machine → ledger entries → outbox → webhook.
- Hours later: reconciliation against processors' settlement files (T+1), and settlement to the merchant from escrow (T+2 by default), both driven by the ledger.
"I'll structure it around one rule: the payment state machine and the ledger are the only truth. Everything else, checkout, routing, processors, webhooks, either feeds them or reads from them. Processors are untrusted for timing, so their answers arrive as events, and anything we don't hear about gets reconciled."
System Architecture#
%% title: Payment gateway architecture
flowchart TD
subgraph L1["Layer 1 · Merchants and customers"]
direction LR
MS@{ icon: "sd:server", label: "Merchant server<br/>orders, capture, refunds" }
CK@{ icon: "sd:phone", label: "Checkout<br/>web / app SDK" }
end
GW@{ icon: "sd:gateway", label: "API gateway<br/>auth, idempotency, rate limit per merchant" }
subgraph L3["Layer 3 · Payment core"]
direction LR
PAY["Payment service<br/>state machine, one row per payment"]:::hot
RT["Router<br/>success rates per method, bank, processor"]:::core
RISK["Risk checks<br/>velocity, device, amount"]:::service
end
subgraph L4["Layer 4 · Processors"]
direction LR
VAULT@{ icon: "sd:lock", label: "Card vault<br/>PCI-DSS, tokens" }
ADP["Processor adapters<br/>acquirers, UPI switch, net banking"]:::service
BANKS@{ icon: "sd:bank", label: "Banks, networks, NPCI" }
end
subgraph L5["Layer 5 · Money and events"]
direction LR
LED@{ icon: "sd:database", label: "Ledger<br/>double-entry, append-only" }
BUS@{ icon: "tech:kafka", label: "Kafka<br/>payment events (outbox)" }
REC["Reconciler<br/>status checks, settlement files"]:::core
end
subgraph L6["Layer 6 · Out to merchants"]
direction LR
WH@{ icon: "sd:webhook", label: "Webhook delivery<br/>queue per merchant, retries 24 h" }
SET["Settlement<br/>escrow → merchant, T+2"]:::service
ESC@{ icon: "sd:bank", label: "Escrow bank" }
end
MS ==> GW
CK ==> GW
GW ==> PAY
PAY --> RISK
PAY ==> RT
RT ==> ADP
CK -.->|"card details"| VAULT
ADP ==> BANKS
PAY ==> LED
PAY -.-> BUS
BUS --> WH
BUS --> REC
LED --> SET --> ESC
class MS,CK client
class GW edge
class VAULT edge
class BANKS,ESC external
class LED store
class BUS queue
class WH queueKEY INSIGHTS:
- The payment service owns one row per payment and is the only thing that moves it between states. Processor callbacks, status checks and merchant calls all become state-transition requests it can accept or refuse
- The card vault is a separate, minimal system: checkout sends card details straight to it and gets a token; the rest of the gateway never sees a card number
- Every state change and its ledger entries commit together, and leave through an outbox onto Kafka (Razorpay's engineers describe the same outbox pattern in their UPI switch); webhooks and reconciliation are consumers
Data Flow: One Card Payment#
"Let's follow Meera paying ₹1,999 at a store that auto-captures."
%% title: One card payment, order to webhook sequenceDiagram autonumber participant M as Merchant server participant G as Gateway participant V as Vault participant R as Router participant B as Acquirer + issuer participant W as Webhooks M->>G: Create order ₹1,999 (idem key o-77) G-->>M: order_A1 Note over G: Meera opens checkout, enters card G->>V: Card details V-->>G: token tk_9 G->>R: Route card, issuer bank X R-->>G: Acquirer P2 (best success rate now) G->>B: Authorise tk_9 ₹1,999 B-->>G: Needs OTP, then authorised Note over G: payment pay_B1: CREATED → AUTHORIZED → CAPTURED · ledger entries in the same commit G-)W: payment.captured, order.paid (event ids) W->>M: POST webhook, signed M-->>W: 200 in 120 ms
MEANWHILE, IN PARALLEL:
- The router's success-rate window for "card, issuer X, acquirer P2" just got one more success
- The reconciler will see pay_B1 again tomorrow, in P2's settlement file, and match it against the ledger
- On day T+2, pay_B1's net amount is part of the store's settlement from escrow
High-level design
Q1/3 Why does checkout send card details straight to the vault instead of through the payment service?
Answer & explanation
Answer: B
The rest of the gateway works with tokens. A breach or a log line elsewhere can then never leak card numbers, and only the vault carries the heaviest compliance burden.
Q2/3 A processor's callback says "success" for a payment the gateway already marked failed after a timeout. Who decides what happens?
Answer & explanation
Answer: C
Only the payment service moves a payment between states. A late success on a failed payment becomes a defined path (capture if the order still wants it, otherwise refund), never an overwrite.
Q3/3 Why do webhooks and reconciliation consume events from Kafka rather than being called directly by the payment service?
Answer & explanation
Answer: C
If the payment service called each consumer, a slow merchant or a crashed reconciler would block or lose events. The outbox plus consumers turns that into durable, independent work.
Phase 4: Deep Dives (20 minutes)#
Deep Dive 1: Idempotency and the Payment State Machine (Week 2 Concepts: Idempotency, Week 5: Consistency Models)#
"'Exactly once' in payments is two promises: each request has one effect, however often it's sent, and each payment has one final state, however many messages arrive about it."
The merchant retried
A merchant's server sends "create order ₹1,999" and its connection drops before the reply. It retries with the same idempotency key. Meanwhile, a different bug in the merchant's code sends the same key with ₹2,999. And on our side, a processor's success callback arrives twice. What happens in each case?
One strong answer
- Same key, same body: the first request is stored with its key and its response; the retry returns that stored response. One order.
- Same key, different body: the key is bound to a hash of the request body; a mismatch is rejected with an error. Silently returning the ₹1,999 order would hide a merchant bug; creating a second order would break the key's meaning.
- A retry while the first is still running: the key is locked while in flight; the retry waits briefly or gets "in progress, retry later", never a second execution.
- Duplicate callback: the transition is guarded by the current state:
AUTHORIZED → CAPTUREDonly applies if the row is inAUTHORIZED. The second callback finds itCAPTUREDand is acknowledged without effect.
The Challenge:
- Merchants retry on timeouts, and their HTTP clients retry automatically
- Customers tap "Pay" twice, or reopen a payment page from history
- Processors resend callbacks; UPI results can arrive as a callback and as a status-check answer
- Our own workers crash between doing something and recording that they did it
The Solution:
%% title: Payment state machine stateDiagram-v2 [*] --> CREATED CREATED --> PENDING: sent to processor PENDING --> AUTHORIZED: bank approved PENDING --> FAILED: bank declined PENDING --> UNKNOWN: no answer in time UNKNOWN --> AUTHORIZED: status check or callback says yes UNKNOWN --> FAILED: says no, or settlement file has no record AUTHORIZED --> CAPTURED: merchant or auto-capture AUTHORIZED --> AUTO_REFUNDED: not captured in 3 days CAPTURED --> REFUNDED: merchant refunds FAILED --> LATE_SUCCESS: money moved after all LATE_SUCCESS --> CAPTURED: order still open LATE_SUCCESS --> AUTO_REFUNDED: order closed CREATED: CREATED · idempotency key stored UNKNOWN: UNKNOWN · reconcile, max 3 UPI checks AUTHORIZED: AUTHORIZED · funds blocked CAPTURED: CAPTURED · ledger entries posted class CREATED,PENDING,AUTHORIZED core class CAPTURED ok class FAILED,REFUNDED,AUTO_REFUNDED bad class UNKNOWN,LATE_SUCCESS queue
KEY DESIGN DECISIONS:
- AN IDEMPOTENCY KEY IS A RECORD, NOT A CACHE ENTRY — Key, merchant, request hash, status (in progress / done) and the stored response, in the same database as the effect, so "done" and "the order exists" commit together.
- TRANSITIONS ARE CONDITIONAL UPDATES —
UPDATE payments SET state = 'CAPTURED' WHERE id = ? AND state = 'AUTHORIZED'. Row count 1 means this request did it; 0 means someone else already did. - UNKNOWN IS A STATE — A timeout never becomes "failed" by itself. It becomes
UNKNOWNwith a reconciliation schedule, and the customer sees "we're confirming your payment". - LATE SUCCESS HAS A PATH — If the money moved after we called a payment failed, it goes to
LATE_SUCCESSand then to capture or automatic refund, depending on whether the merchant's order is still open. - AUTHORISED IS NOT FOREVER — Following the documented rule, an authorised payment that isn't captured within 3 days is refunded automatically.
# api/idempotency.py
"""
Idempotent write APIs. The key, a hash of the request, and the stored
response live in the same database as the effect, written in the same
transaction, so a crash can never leave "effect done, key not recorded".
"""
import hashlib
import json
from dataclasses import dataclass
from typing import Callable, Optional
@dataclass
class Stored:
status: str # IN_PROGRESS | DONE
request_hash: str
response: Optional[dict]
class IdempotencyMismatch(Exception):
pass
class InProgress(Exception):
pass
def request_hash(method: str, path: str, body: dict) -> str:
return hashlib.sha256(f"{method} {path} {json.dumps(body, sort_keys=True)}".encode()).hexdigest()
class IdempotentHandler:
def __init__(self, db):
self.db = db
def run(self, merchant_id: str, key: str, method: str, path: str, body: dict,
effect: Callable[[object], dict]) -> dict:
h = request_hash(method, path, body)
with self.db.transaction() as tx:
inserted = tx.execute(
"INSERT INTO idem_keys (merchant_id, key, request_hash, status) "
"VALUES (%s, %s, %s, 'IN_PROGRESS') ON CONFLICT DO NOTHING",
(merchant_id, key, h),
).rowcount
if inserted == 0:
row: Stored = tx.fetch_one(
"SELECT status, request_hash, response FROM idem_keys "
"WHERE merchant_id = %s AND key = %s FOR UPDATE", (merchant_id, key))
if row.request_hash != h:
raise IdempotencyMismatch(key) # same key, different request
if row.status == "DONE":
return row.response # a retry: same answer
raise InProgress(key) # first attempt still running
response = effect(tx) # creates the order, in this tx
tx.execute(
"UPDATE idem_keys SET status = 'DONE', response = %s "
"WHERE merchant_id = %s AND key = %s",
(json.dumps(response), merchant_id, key),
)
return response
def transition(tx, payment_id: str, frm: str, to: str) -> bool:
"""The only way a payment changes state. False means someone else got there first."""
return tx.execute(
"UPDATE payments SET state = %s, updated_at = now() WHERE id = %s AND state = %s",
(to, payment_id, frm),
).rowcount == 1"The effect inside run only touches our own database. Calls to processors are never made inside that transaction: the payment is created first, then the processor call happens, and its result comes back as a transition. That's how a crash halfway through leaves a payment in PENDING, which reconciliation knows how to finish, instead of a charge nobody recorded."
Deep Dive 2: Routing to the Processor Most Likely to Say Yes (Week 2 Concepts: Circuit Breakers, Week 10: Observability)#
"The same card payment can often go through several acquiring banks. On a good day they're similar; on a bad day one of them, or one issuing bank behind it, is failing half its payments. Routing is where the gateway turns that into a higher success rate for merchants."
One route is failing
For Visa cards from issuer X, acquirer P1 normally approves 92% and P2 89%. In the last five minutes, P1's success rate for that slice dropped to 60%, but only for issuer X. What should the router do, how quickly, and how does it know when P1 has recovered?
One strong answer
- Measure per slice, not per processor: keep success rates per (method, network, issuer, processor) over a short sliding window, because failures are usually local: one issuer through one acquirer.
- Move traffic fast: when a slice drops well below its normal rate, with enough attempts to be sure, mark that route down for that slice and send its traffic to the next-best processor. Razorpay's documentation for its Optimizer product describes this behaviour: a temporary downtime of 20 minutes when a high-priority gateway's success rate drops, with traffic moving to the next priority.
- Keep probing: while a route is down, send a small share of traffic (say 2%) to it to learn when it recovers, and bring it back gradually.
- Separate customer failures from route failures: a wrong OTP or insufficient funds is the customer's answer, not the processor's fault; count only technical failures and bank-side declines that differ between routes.
- Respect merchant rules: some merchants pin processors for contract or pricing reasons; route within what they allow.
The Challenge:
- An issuer's authentication page slows down, and every payment sent there through one acquirer times out
- A processor's overall success rate looks fine while one bank behind it fails half the time
- Switching on noise sends traffic back and forth every minute
- A route that is never tried again after an outage is never trusted again
The Solution:
%% title: The routing decision for one payment
flowchart TD
P["Payment<br/>card · Visa · issuer X · ₹1,999"]:::core
ALLOW["Merchant's allowed processors<br/>and priorities"]:::service
SR@{ icon: "sd:chart", label: "Success rate per slice<br/>last 5 min + 1 h baseline" }
DOWN{"Route marked down<br/>for this slice?"}:::cache
PICK["Pick best score<br/>success rate, cost, latency"]:::hot
PROBE["2% probe traffic<br/>to down routes"]:::service
ADP["Processor adapter"]:::service
P --> ALLOW --> DOWN
SR --> DOWN
DOWN -->|"no"| PICK
DOWN -->|"yes, probe share"| PROBE
PICK ==> ADP
PROBE --> ADP
ADP -.->|"result"| SR
class SR cache# routing/router.py
"""
Route a payment to the processor most likely to approve it, using
success rates per slice (method, network, issuer, processor). A slice
that drops far below its baseline is marked down for a while; a small
probe share keeps testing it so it can come back.
"""
import random
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, Dict, List, Optional, Tuple
WINDOW_S = 300
MIN_ATTEMPTS = 50
DOWN_FOR_S = 20 * 60 # the documented Optimizer behaviour uses 20 minutes
PROBE_SHARE = 0.02
@dataclass
class SliceStats:
events: Deque[Tuple[float, bool]] = field(default_factory=deque) # (ts, technical_ok)
baseline: float = 0.9
down_until: float = 0.0
def rate(self, now: float) -> Optional[float]:
while self.events and self.events[0][0] < now - WINDOW_S:
self.events.popleft()
if len(self.events) < MIN_ATTEMPTS:
return None
return sum(ok for _, ok in self.events) / len(self.events)
class Router:
def __init__(self, rng: random.Random):
self.stats: Dict[Tuple[str, str], SliceStats] = {}
self.rng = rng
def record(self, slice_key: str, processor: str, technical_ok: bool, now: float) -> None:
"""Called with each result. Customer declines (wrong OTP, no funds) are not recorded."""
s = self.stats.setdefault((slice_key, processor), SliceStats())
s.events.append((now, technical_ok))
r = s.rate(now)
if r is not None and r < s.baseline - 0.15:
s.down_until = now + DOWN_FOR_S # move traffic away, for a while
def choose(self, slice_key: str, allowed: List[Tuple[str, int]], now: float) -> str:
"""allowed: (processor, merchant priority), lower priority number first."""
up, down = [], []
for proc, prio in sorted(allowed, key=lambda a: a[1]):
s = self.stats.get((slice_key, proc))
(down if s and s.down_until > now else up).append((proc, prio, s))
if down and (not up or self.rng.random() < PROBE_SHARE):
return down[0][0] # keep learning about down routes
if not up:
return sorted(allowed, key=lambda a: a[1])[0][0]
def score(item):
proc, prio, s = item
r = (s.rate(now) if s else None) or (s.baseline if s else 0.9)
return r - prio * 0.01 # merchant priority breaks near-ties
return max(up, key=score)[0]Idempotency and routing
Q1/4 A merchant sends the same idempotency key twice, the second time with a different amount. What should the API do?
Answer & explanation
Answer: C
A key means "this exact request". A different body under the same key is a bug on the merchant's side, and both silent options would hide it.
Q2/4 Two success callbacks arrive for the same payment. Why is the second one harmless?
Answer & explanation
Answer: B
UPDATE ... WHERE state = 'AUTHORIZED'affects one row the first time and none the second. The duplicate is acknowledged with no effect.Q3/4 Acquirer P1's success rate drops to 60% for issuer X only. Why keep stats per (method, network, issuer, processor) rather than per processor?
Answer & explanation
Answer: B
A per-processor average would barely move, or would wrongly move all traffic off P1. Slicing finds the failing route and moves only that traffic.
Q4/4 Why send ~2% of traffic to a route that is marked down?
Answer & explanation
Answer: C
Without probes, a route that failed once is never measured again. A small share tells the router when the success rate is back to its baseline.
Deep Dive 3: The Bank Didn't Answer, Reconciling Unknown Outcomes (Week 5 Concepts: Saga Orchestration, Week 3: Audit Logs)#
"Every payment system ends up here. The customer approved the payment in their UPI app, and we never heard back. The money may have moved, or not. We must find out, without asking too often and without ever charging twice."
The UPI result never came
Meera approves a UPI payment. Thirty seconds later there's no callback. NPCI allows three status checks, at least 90 seconds apart. What do you show Meera and the merchant now, what do you do over the next few minutes, and what's your final safety net if the checks don't settle it?
One strong answer
- Now: the payment goes to
UNKNOWN. Meera sees "we're confirming your payment, don't pay again"; the merchant's order stays "attempted", not "failed", and the merchant is told not to ship and not to ask for a new payment yet. - Next few minutes: a callback may still arrive and settles it. If not, status checks at about 90 s, 180 s and 270 s after the payment, each keyed on the same transaction reference. A definite answer moves the payment to
AUTHORIZED(then capture) orFAILED. - If still unknown: wait for the processor's settlement or MIS file (T+1). A payment in the file is a success; a payment missing from it is a failure.
- If money moved after we said "failed": the payment goes to
LATE_SUCCESS: captured if the merchant's order is still open, otherwise refunded automatically, and both sides are told. - Never: create a new payment for the same order while one is
UNKNOWN, or turn a timeout into "failed" by itself.
The Challenge:
- NPCI allows a UPI status check only three times, 90 seconds apart, since August 2025
- Banks answer status checks from the same systems that are slow right now
- Customers retry with a new payment when they see "failed", so a premature "failed" creates double charges
- Some answers only exist in the next day's settlement files
The Solution:
%% title: Resolving an unknown UPI payment
sequenceDiagram
autonumber
participant G as Payment service
participant S as UPI switch / processor
participant R as Reconciler
participant F as Settlement files
G->>S: Collect / intent request (txn ref T1)
Note over G: 30 s, no callback · state UNKNOWN · customer told "confirming"
R->>S: Status check 1 at 90 s (T1)
S-->>R: Pending
R->>S: Status check 2 at 180 s (T1)
alt definite answer
S-->>R: Success
R->>G: UNKNOWN → AUTHORIZED, then capture
else still pending after check 3
R->>F: Next day: is T1 in the settlement file?
F-->>R: T1 present
R->>G: Late success: capture if order open, else refund
end# recon/reconciler.py
"""
Drives every UNKNOWN payment to a final state. Callbacks are handled
elsewhere; this worker covers the payments we heard nothing about,
within processors' limits on status checks, and falls back to
settlement files. Every action is a guarded transition.
"""
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional
UPI_CHECK_AT = [timedelta(seconds=90), timedelta(seconds=180), timedelta(seconds=270)]
@dataclass
class Payment:
id: str
order_id: str
method: str
processor: str
processor_ref: str
state: str
created_at: datetime
checks_done: int = 0
class Reconciler:
def __init__(self, db, adapters, orders, scheduler, notifier):
self.db = db
self.adapters = adapters
self.orders = orders
self.scheduler = scheduler
self.notifier = notifier
def on_unknown(self, p: Payment) -> None:
for delay in UPI_CHECK_AT if p.method == "upi" else [timedelta(seconds=30), timedelta(minutes=5)]:
self.scheduler.at(p.created_at + delay, "status_check", p.id)
def status_check(self, payment_id: str) -> None:
p = self.db.get_payment(payment_id)
if p.state != "UNKNOWN":
return # a callback already settled it
answer = self.adapters[p.processor].status(p.processor_ref) # same reference every time
self.db.increment_checks(p.id)
if answer == "SUCCESS":
self._settle(p, success=True)
elif answer == "FAILED":
self._settle(p, success=False)
# "PENDING": wait for the next scheduled check, or the settlement file
def on_settlement_file_row(self, processor_ref: str, amount_paise: int) -> None:
p = self.db.get_payment_by_ref(processor_ref)
if p.state == "UNKNOWN":
self._settle(p, success=True)
elif p.state == "FAILED":
self._late_success(p) # money moved after we said no
# CAPTURED / REFUNDED: matched, nothing to do except mark reconciled
def close_unmatched(self, file_date: datetime) -> None:
"""After the file for a day is complete, UNKNOWN payments not in it failed."""
for p in self.db.unknown_created_before(file_date):
self._settle(p, success=False)
def _settle(self, p: Payment, success: bool) -> None:
with self.db.transaction() as tx:
if success:
if self.db.transition(tx, p.id, "UNKNOWN", "AUTHORIZED"):
self.db.outbox(tx, "payment.authorized", p.id)
elif self.db.transition(tx, p.id, "UNKNOWN", "FAILED"):
self.db.outbox(tx, "payment.failed", p.id)
def _late_success(self, p: Payment) -> None:
with self.db.transaction() as tx:
if not self.db.transition(tx, p.id, "FAILED", "LATE_SUCCESS"):
return
if self.orders.is_open(p.order_id, tx=tx):
self.db.transition(tx, p.id, "LATE_SUCCESS", "CAPTURED")
self.db.outbox(tx, "payment.captured", p.id)
else:
self.db.transition(tx, p.id, "LATE_SUCCESS", "AUTO_REFUNDED")
self.db.outbox(tx, "refund.created", p.id)
self.notifier.customer_and_merchant(p.id)"The reconciler also runs the other way every day: for every row in every processor's settlement file, there must be a ledger entry with the same reference and amount, and for every captured payment, a file row. Anything unmatched is a ticket for a person, not something the code guesses about."
Deep Dive 4: The Ledger, Escrow and Settlement (Week 3 Concepts: Audit Log System, Week 5: Consistency)#
"Payment states say what happened to a payment. The ledger says where every rupee is. They are different questions, and the second one is what auditors, regulators and merchants care about."
Where is the money right now?
Meera's ₹1,999 is captured at 10:00 on Monday. The gateway's fee is 2% plus 18% GST on the fee. On Tuesday the merchant refunds ₹500. Settlement is T+2 working days. Which accounts move, by how much, and how do you prove nothing was lost?
One strong answer
- On capture (Monday): the money is owed to us by the processor until it reaches escrow. Entries: debit "processor receivable" ₹1,999; credit "merchant payable" ₹1,999. Then the fee: debit "merchant payable" ₹47.18; credit "fee revenue" ₹39.98 and "GST payable" ₹7.20.
- When the processor's funds reach escrow: debit "escrow cash" ₹1,999; credit "processor receivable" ₹1,999.
- On refund (Tuesday): debit "merchant payable" ₹500; credit "refund payable to customer" ₹500, then cleared when the refund goes out.
- On settlement (Wednesday, T+2): pay the merchant what is left: ₹1,999 − ₹47.18 − ₹500 = ₹1,451.82. Debit "merchant payable"; credit "escrow cash".
- Proof: every posting is a set of entries that sums to zero; the ledger is append-only; the escrow account's bank statement must equal the "escrow cash" balance every day. A difference is an incident, not a rounding error.
The Challenge:
- Updating a merchant's "balance" in place loses why it changed, and two concurrent updates lose one of them
- Refunds, chargebacks and fees arrive on different days from the payment
- The bank, the processor and our records each have their own view of the same money
- An aggregator must keep merchants' funds in escrow, so our numbers must match a bank statement every day
The Solution:
%% title: Money from customer to merchant
flowchart TD
CUS@{ icon: "sd:user", label: "Customer's bank" }
PRC@{ icon: "sd:bank", label: "Processor / acquirer" }
ESC@{ icon: "sd:bank", label: "Escrow account<br/>scheduled commercial bank" }
LED["Ledger<br/>double-entry, append-only"]:::hot
SET["Settlement run<br/>T+2, or instant"]:::core
MER@{ icon: "sd:building", label: "Merchant's bank account" }
CUS ==>|"₹1,999"| PRC
PRC ==>|"funds, T+1"| ESC
ESC ==>|"₹1,451.82"| MER
LED -.->|"payable per merchant"| SET
SET ==> ESC
class CUS client
class PRC,MER external
class ESC store# ledger/postings.py
"""
Double-entry ledger. Every business event is one posting: a set of
entries that sums to zero, written in one transaction and never
updated. Balances are sums of entries (cached, but always rebuildable).
Amounts are integer paise; never floats.
"""
from dataclasses import dataclass
from typing import List
FEE_BPS = 200 # 2% (illustrative pricing)
GST_BPS = 1800 # 18% on the fee
@dataclass(frozen=True)
class Entry:
account: str # e.g. "merchant_payable:m_42"
amount_paise: int # debit positive, credit negative
class UnbalancedPosting(Exception):
pass
class Ledger:
def __init__(self, db):
self.db = db
def post(self, tx, posting_id: str, kind: str, entries: List[Entry]) -> None:
if sum(e.amount_paise for e in entries) != 0:
raise UnbalancedPosting(posting_id)
# posting_id is unique: replaying the same event can never post twice
tx.execute("INSERT INTO postings (id, kind) VALUES (%s, %s)", (posting_id, kind))
for e in entries:
tx.execute("INSERT INTO entries (posting_id, account, amount_paise) VALUES (%s, %s, %s)",
(posting_id, e.account, e.amount_paise))
def capture_entries(merchant: str, processor: str, amount: int) -> List[Entry]:
fee = amount * FEE_BPS // 10_000
gst = (fee * GST_BPS + 5_000) // 10_000 # rounded to the nearest paisa
return [
Entry(f"processor_receivable:{processor}", amount),
Entry(f"merchant_payable:{merchant}", -amount),
Entry(f"merchant_payable:{merchant}", fee + gst),
Entry("fee_revenue", -fee),
Entry("gst_payable", -gst),
]
def settlement_entries(merchant: str, payable: int) -> List[Entry]:
return [
Entry(f"merchant_payable:{merchant}", payable),
Entry("escrow_cash", -payable),
]A few details carry the correctness:
- The capture's state transition and its posting commit in the same transaction as the outbox event, so a payment is never
CAPTUREDwithout its entries. - Postings are keyed (
capture:pay_B1,refund:rf_3): a replayed event inserts nothing, because the posting id already exists. - Settlement reads balances, not payments: the amount to pay a merchant is the balance of
merchant_payable:m_42for entries older than the cut-off, which already includes fees, refunds and chargebacks. - Every day, three numbers must agree: the ledger's
escrow_cash, the escrow bank's statement, and the sum of processors' settlement files minus payouts.
Reconciliation and the ledger
Q1/4 A UPI payment has no callback after 30 seconds. Why not mark it failed and let the customer try again?
Answer & explanation
Answer: C
A timeout is "unknown". Calling it failed invites a second payment for the same order. UNKNOWN, honest messaging and reconciliation avoid the double charge.
Q2/4 A payment we marked FAILED appears in the next day's settlement file. The merchant's order is closed. What happens?
Answer & explanation
Answer: C
The settlement file is the final word on whether money moved. With the order closed, the only right outcome is an automatic refund.
Q3/4 Why keep a double-entry, append-only ledger instead of a balance column per merchant?
Answer & explanation
Answer: B
In-place balances lose history and can lose concurrent updates. Postings that sum to zero, never edited, make every rupee traceable and every balance provable.
Q4/4 Meera's ₹1,999 is captured with a ₹47.18 fee (2% plus 18% GST), then ₹500 is refunded. How much is settled to the merchant at T+2?
Answer & explanation
Answer: D
₹1,999 − ₹47.18 − ₹500 = ₹1,451.82. Settlement pays the merchant_payable balance, which already includes fees and refunds.
Deep Dive 5: Telling the Merchant, at Least Once (Week 2 Concepts: Webhook Delivery, Week 9: Noisy Neighbour)#
"Webhooks are where our exactly-once state meets the merchant's code. We can't make delivery exactly once over the internet; we make it at least once and give the merchant what they need to deduplicate."
The merchant's server is down
A large merchant's server starts returning 500s at 9 PM during a sale. Its webhooks pile up: thousands a minute. Meanwhile, ten thousand small merchants are receiving theirs normally. How do you keep retrying the big merchant without hurting anyone else, and what does the merchant's code need to do?
One strong answer
- A queue per merchant: each merchant's deliveries are independent, with their own concurrency limit. A failing merchant's retries back up in its own queue, not in a shared one.
- Retry with exponential backoff for 24 hours: the documented behaviour is retries for 24 hours with exponential backoff, a 5-second response limit, and the webhook disabled after 24 hours of continuous failure, with the merchant alerted.
- Every event has an id and a signature: the merchant verifies the signature (HMAC over the body with their secret) and deduplicates on the event id, because the same event can arrive more than once.
- Order isn't guaranteed: the merchant must not assume
payment.capturedarrives afterpayment.authorized; it should treat events as hints and fetch the payment's current state when it matters. - Answer fast, work later: the merchant's handler should store the event and return 200 within 5 seconds; slow handlers time out and cause duplicates.
The Solution:
%% title: Webhook delivery, isolated per merchant
flowchart TD
BUS@{ icon: "tech:kafka", label: "Kafka<br/>payment events" }
FAN["Fan-out<br/>event → subscribed merchants"]:::service
MQ@{ icon: "sd:queue", label: "Queue per merchant<br/>concurrency limit each" }
SEND["Sender<br/>sign · POST · 5 s timeout"]:::hot
RETRY@{ icon: "sd:timer", label: "Retry schedule<br/>exponential backoff, 24 h" }
MS@{ icon: "sd:server", label: "Merchant server<br/>verify, dedupe on event id" }
OFF["Disable after 24 h of failures<br/>alert the merchant"]:::bad
BUS ==> FAN ==> MQ ==> SEND ==> MS
SEND -.->|"non-2xx or timeout"| RETRY
RETRY -.-> MQ
RETRY -.->|"24 h"| OFF
class BUS queue
class MS client
class RETRY cache# webhooks/sender.py
"""
At-least-once webhook delivery. Each merchant has its own queue and
concurrency limit, so one failing merchant can't slow the others.
Every delivery is signed and carries a stable event id.
"""
import hashlib
import hmac
import json
import random
from dataclasses import dataclass
TIMEOUT_S = 5
RETRY_WINDOW_S = 24 * 3600
BASE_S, CAP_S = 30, 3600
@dataclass
class Delivery:
event_id: str # stable across retries: the merchant dedupes on it
merchant_id: str
url: str
body: dict
created_at: float
attempt: int = 0
def signature(secret: bytes, raw: bytes) -> str:
return hmac.new(secret, raw, hashlib.sha256).hexdigest()
def next_delay(attempt: int, rng: random.Random) -> float:
"""Exponential backoff with jitter, capped, so retries don't arrive in waves."""
return min(CAP_S, BASE_S * 2 ** attempt) * (0.5 + rng.random() / 2)
class Sender:
def __init__(self, http, secrets, queues, alerts, rng):
self.http = http
self.secrets = secrets
self.queues = queues # per-merchant queues with concurrency limits
self.alerts = alerts
self.rng = rng
def deliver(self, d: Delivery, now: float) -> None:
raw = json.dumps(d.body, separators=(",", ":")).encode()
headers = {
"X-Event-Id": d.event_id,
"X-Signature": signature(self.secrets[d.merchant_id], raw),
}
try:
r = self.http.post(d.url, data=raw, headers=headers, timeout=TIMEOUT_S)
if 200 <= r.status < 300:
return # done; never resent
except TimeoutError:
pass # may have been processed: hence event ids
if now - d.created_at >= RETRY_WINDOW_S:
self.queues.disable(d.merchant_id)
self.alerts.merchant(d.merchant_id, "webhook disabled after 24 hours of failures")
return
d.attempt += 1
self.queues.schedule(d.merchant_id, d, run_at=now + next_delay(d.attempt, self.rng))Webhooks
Q1/3 Why give each merchant its own webhook queue and concurrency limit?
Answer & explanation
Answer: B
Isolation per merchant turns one merchant's outage into that merchant's backlog. Everyone else keeps receiving events on time.
Q2/3 A merchant's handler takes 8 seconds to process each webhook before answering. What will happen?
Answer & explanation
Answer: C
The response limit is 5 seconds. The fix on the merchant's side is to store the event, answer 200 at once, and process it afterwards, deduplicating on the event id.
Q3/3 The merchant receives payment.captured before payment.authorized. What should its code do?
Answer & explanation
Answer: D
Ordering isn't guaranteed. The payment's state in the API is the truth; the webhook says "something changed, go look".
Phase 5: Scaling and Edge Cases (5 minutes)#
"Walk me through a sale midnight, then tell me what breaks."
A Sale Midnight#
- Days before · Plan
- The merchant shares expected volume; capacity for its slices is reserved with processors
- Its traffic gets its own rate limit and connection pools, so its spike can't starve other merchants
- 11:50 PM · Warm up
- Router stats are fresh; probe traffic confirms every route is healthy
- Deployments frozen; on-call watching the sale dashboard
- 00:00 · The spike
- Payments jump to many times normal for this merchant in seconds; the gateway's own work stays under 100 ms
- Unknowns rise as some banks slow; the reconciler's schedule absorbs them
- 00:15 · Card authentication slows at one issuer
- The router moves that issuer's traffic to the next-best acquirer for that slice; probes keep testing
- Next day · Reconcile and settle
- Settlement files matched against the ledger; unknowns closed; late successes refunded or captured
- At T+2, the merchant's settlement includes the night's captures minus fees and refunds
Edge Cases#
Edge case 1: Customer pays twice for one order
Problem: The first UPI payment was slow, so Meera paid again from another app; both succeeded.
Solution:
- The order accepts one captured payment; the second success is captured by the state machine as a duplicate for that order and refunded automatically
- While a payment is
UNKNOWN, checkout says "we're confirming your payment, don't pay again" and offers no second attempt
Edge case 2: Callback arrives before our own record
Problem: A processor's success callback arrives a few milliseconds before the payment row's PENDING commit is visible.
Solution:
- Callbacks for unknown or not-yet-pending payments are stored and retried for a short window, never dropped
- The transition is guarded by state, so applying it late is safe
Edge case 3: A processor goes down completely
Problem: One acquirer stops answering at all.
Solution:
- Its success rate falls to zero within seconds; the router marks every slice down and moves traffic to others
- Payments already sent to it are
UNKNOWNand reconciled from its status API or files once it's back
Edge case 4: Chargeback weeks later
Problem: A cardholder disputes a payment a month after settlement.
Solution:
- A posting moves the amount from the merchant's payable (or a reserve) to "chargeback receivable"; the merchant is notified with evidence deadlines
- If the merchant's balance is too low, future settlements are netted
Edge case 5: Stolen card testing
Problem: A script tries thousands of stolen card numbers at ₹1 on one small merchant.
Solution:
- Velocity checks per card fingerprint, device, IP and merchant; challenge or block before the processor sees them
- Under the RBI's authentication rules, issuers also apply their own risk-based checks on top of two-factor authentication
Scaling and edge cases
Q1/3 Meera's first UPI payment was slow, she paid again, and both succeeded. What should happen?
Answer & explanation
Answer: B
An order accepts one captured payment. The state machine recognises the second success for the same order and refunds it, without a support ticket.
Q2/3 A processor's success callback arrives before the payment's PENDING state is committed. What does the gateway do with it?
Answer & explanation
Answer: C
Dropping it would leave the payment unknown for minutes. Holding it briefly and applying the transition once the row is ready costs nothing and loses nothing.
Q3/3 Why give a big merchant's sale its own rate limit and connection pools?
Answer & explanation
Answer: C
One merchant's midnight is everyone else's normal Tuesday. Bulkheads per merchant keep the spike inside its own capacity.
Phase 6: Monitoring and Operations#
"In payments, the most important metric isn't latency. It's whether every rupee is accounted for: success rate by slice, unknowns by age, and the daily three-way match."
Monitoring Dashboard#
Throughput and success
| Metric | Now |
|---|---|
| Payments a second (all merchants) | 3,840 |
| Success rate, cards / UPI / net banking | 91.2% / 93.8% / 88.5% |
| Gateway's own latency p99 | 64 ms |
| Payments in UNKNOWN (oldest) | 2,310 (4 min 10 s) |
Routing
- Slices marked down: 3 (issuer X via acquirer P1, two net-banking banks)
- Probe traffic: 2% to down routes
Webhooks
- Delivered within 5 s of the event: 99.4%
- Merchants in backoff: 37; largest backlog: 18,200 events (one merchant returning 500s)
Money
- Yesterday's three-way match (ledger, escrow statement, files): matched, 0 open items
- Late successes today: 412, all captured or refunded
Alerts
- 00:00:40 (amber) — Card success for issuer X via P1 at 61%, route down for 20 minutes
- 00:03:12 (amber) — Merchant m_1180 webhook failures at 100%, retries backing off
- 00:05:00 (green) — UNKNOWN count steady; reconciliation keeping up
SLOs for the Gateway#
SLO 1: Exactly one outcome
- Target: 100% of payments reach a final state; none in UNKNOWN beyond the next settlement file
- Error budget: 0
SLO 2: Money integrity
- Target: ledger, escrow statement and processors' files match every day; open items resolved within 2 working days
SLO 3: Availability
- Target: 99.99% of payment API requests succeed (excluding customer declines)
SLO 4: Merchant notification
- Target: 99% of webhooks sent within 5 s of the state change; every event retried for 24 hours
SLO 5: Safe change
- Target: every deployment canaried and judged before reaching the most critical merchants
"On that last SLO, Razorpay's engineering team has written about how they deploy: canaries that step through 1%, 2%, 5%, 10% and 15% of traffic with automated analysis at each step, and rollouts ordered by merchant criticality so the most critical merchants see a build last. That's the right instinct for a payment system: most incidents start with a change, not a crash."
Reading the dashboard
Q1/3 2,310 payments are in UNKNOWN, the oldest 4 minutes 10 seconds. Is that an incident?
Answer & explanation
Answer: A
UNKNOWN is a normal state at peak. The alarm is the age and the trend: payments stuck past the last status check, or a count that keeps growing.
Q2/3 One merchant has 18,200 webhook events backed up. What is the effect on other merchants?
Answer & explanation
Answer: B
Isolation per merchant keeps one merchant's outage inside its own backlog. Retries back off for 24 hours, then the webhook is disabled and the merchant alerted.
Q3/3 Why is the daily three-way match an SLO with an error budget of zero open items left unresolved?
Answer & explanation
Answer: C
The ledger, the escrow bank's statement and processors' files are three views of the same money. When they agree, nothing was lost; when they don't, a person investigates.
Final Challenge: Midnight Sale, One Slow Bank#
The interviewer leans back. "Let's put it all together."
"A large merchant's sale opens at midnight with twenty times its normal traffic. At 00:02, the card authentication pages of one big issuing bank slow to 20 seconds, and its UPI responses start timing out. Thousands of customers are mid-payment. Merchants' servers are getting hammered too. What do you do?"
- One merchant at 20× normal; everyone else normal
- One issuing bank slow for cards (20 s authentication) and timing out on UPI
- Customers retry when they see spinners
- Some merchants' servers return errors under their own load
- Nothing may be charged twice, and every payment must end in one final state
Contain the bank, protect the customer, keep the books
What changes in routing, in checkout, in reconciliation and in webhooks? What do customers of the slow bank see?
One strong answer
- Contain the merchant: its rate limit and pools are its own. Its 20× can use its reserved capacity and no more, so other merchants' payments stay at normal latency.
- Routing can't fix an issuer: for cards from that bank, every acquirer reaches the same slow authentication page, so moving traffic doesn't help. The router learns that (all slices for the issuer are down) and stops flapping between acquirers.
- Tell customers early: checkout shows a notice for that bank ("payments from this bank are slow right now") and suggests another method, before they start. That's cheaper and kinder than a 20-second spinner.
- Unknowns grow, by design: UPI timeouts become UNKNOWN, with three status checks 90 seconds apart and the settlement file as the backstop. Customers see "confirming, don't pay again", and a second attempt for an order with an UNKNOWN payment is blocked.
- Late money is handled: payments that turn out to have succeeded become late successes: captured if the order is open, otherwise refunded automatically.
- Webhooks back off per merchant: merchants whose servers are failing get exponential retries in their own queues; nobody else waits.
- The books still balance: every state change still posts its entries; tomorrow's three-way match proves it.
Final challenge · Quest check
Q1/4 One issuing bank's card authentication is slow for every acquirer. Why doesn't routing help?
Answer & explanation
Answer: B
Routing helps when one path is worse than another. When the issuer itself is slow, the fix is to tell its customers early and suggest another method.
Q2/4 Customers of the slow bank keep tapping "Pay" again. What stops double charges?
Answer & explanation
Answer: C
The order is the unit. While its payment is unknown, checkout says "confirming" and refuses a second attempt; if one slips through and both succeed, the duplicate is refunded.
Q3/4 The sale's merchant is at 20× normal. What protects every other merchant?
Answer & explanation
Answer: C
Bulkheads per merchant keep one sale's spike in its own compartment, for payments and for notifications.
Q4/4 At 10,000 TPS with ~15 internal operations per payment, how many operations a second must the gateway handle?
Answer & explanation
Answer: C
10,000 × 15 = 150,000. Sizing for payments alone would miss the routing, ledger, events and webhooks each payment creates.
Interview Conclusion#
"Good. A few quick ones to finish."
"Why not use a distributed transaction across the gateway, the processor and the bank?"
"Because the bank isn't a participant we can lock. It answers when it answers, and sometimes only in tomorrow's file. So the design is a state machine with guarded transitions, an UNKNOWN state, and reconciliation against independent records. That's how you get exactly one outcome without anyone holding a lock across the internet."
"Where does 'exactly once' really live?"
"In two places we own: the idempotency record that commits with the effect, and the conditional state transition on one row. Everything that crosses a network, callbacks, status checks, webhooks, is at least once, and is made safe by those two."
"What's the single most important design decision?"
"Never turning silence into an answer. A timeout is UNKNOWN, not failed. That one rule is what stops a slow bank from becoming double charges, and the ledger plus the daily match is what proves we got every case right."
Summary: Concepts Applied from 10-Week Course#
- Week 2: Failure-first design
- Idempotency — keys stored with the effect, bound to a request hash
- Timeouts — a timeout becomes UNKNOWN, never "failed"
- Circuit breakers — routes marked down per slice, with probes to recover
- Webhook delivery — at least once, signed, event ids, backoff for 24 hours
- Week 3: Messaging and async processing
- Transactional outbox — state change, ledger entries and event in one commit
- Audit log — an append-only, double-entry ledger
- Week 5: Consistency and coordination
- Conditional state transitions on one row as the only writer
- Saga orchestration — reconciliation drives every payment to a final state
- Week 9: Multi-tenancy and security
- Noisy neighbour — per-merchant limits, pools and webhook queues
- A minimal card vault; tokens everywhere else
- Week 10: Production readiness
- SLOs for money integrity and exactly one outcome; canaried deployments
Why Razorpay Matters#
Exactly once across parties you don't control
- The bank, the network and the merchant each have their own timing and their own records. "Exactly once" is built from idempotency, guarded transitions and reconciliation, not from a transaction
Silence is the normal failure
- At a billion transactions a quarter, some payments always have no answer yet. Designing UNKNOWN as a state, and resolving it within NPCI's limits and through settlement files, is the core of the job
Money must be provable
- The ledger, the escrow statement and processors' files must agree every day. A gateway that can't prove where every rupee is, isn't done
"The bank that never answered isn't the bug. The bug is the system that guessed."
Self-Assessment Checklist#
After studying this case study, you should be able to:
Architecture:
- Draw a gateway with a payment state machine, a vault, a router, adapters, a ledger and an outbox
- Explain which steps are exactly once and which are at least once with deduplication
Correctness:
- Implement idempotent APIs whose key record commits with the effect
- Model UNKNOWN and LATE_SUCCESS states and resolve them by reconciliation
- Explain why status checks can't replace settlement-file reconciliation
Routing:
- Route by success rate per (method, network, issuer, processor) with temporary downtime and probes
Money:
- Post balanced, append-only ledger entries for capture, fees, refunds and settlement
- Compute a merchant's settlement from the ledger and prove it with a three-way match
Operations:
- Deliver webhooks at least once with per-merchant isolation
- Define SLOs for exactly one outcome and money integrity
Sources#
Company figures:
- Read
Business Today — Co-founder and CEO of Razorpay on why it's never okay to be second (over a billion transactions every quarter; annualised payments of more than $180 billion), August 2026: https://www.businesstoday.in/magazine/interview/story/co-founder-and-ceo-of-razorpay-on-why-its-never-okay-to-be-second-546897-2026-08-03
- Read
Entrackr — Razorpay, Cashfree get final PA licence, can onboard new merchants, December 2023: https://entrackr.com/2023/12/razorpay-cashfree-get-final-pa-license-can-onboard-new-merchants
Razorpay engineering and documentation:
- Blog
Kshitij Nawandar, Razorpay Engineering — Five years of Kafka at Razorpay's UPI switch (UPI is asynchronous; a peak of 400 TPS during the IPL on SQS; a 10,000 TPS target for the second version; the outbox pattern; 70%+ of Razorpay's UPI volume), September 2026: https://engineering.razorpay.com/tryst-with-kafka-2f5cef766c45
- Book
Anusha Musthaphak, Razorpay Engineering — How we deploy reliably at Razorpay (canary steps of 1%, 2%, 5%, 10% and 15%; rollouts ordered by merchant criticality), October 2026: https://engineering.razorpay.com/how-we-deploy-reliably-at-razorpay-52ddf676caf8
- Docs
Razorpay Docs — Payment states (created, authorized, captured, refunded, failed; authorised payments auto-refunded if not captured within 3 days): https://razorpay.com/docs/payments/payments/
- Book
Razorpay Docs — Settlements (T+2 working days by default; instant settlements on request): https://razorpay.com/docs/payments/settlements/
- Docs
Razorpay Docs — Webhooks best practices (retries for 24 hours with exponential backoff; 5-second timeout; at-least-once delivery; event id for deduplication; order not guaranteed): https://razorpay.com/docs/webhooks/best-practices/
- Docs
Razorpay Docs — Optimizer dynamic routing (temporary downtime of 20 minutes when a high-priority gateway's success rate drops; fallback to the next priority): https://razorpay.com/docs/payments/optimizer/dynamic-routing.md
Regulation and the payments network:
- Book
Reserve Bank of India — Master Direction on Regulation of Payment Aggregators (escrow with a scheduled commercial bank; ₹15 crore / ₹25 crore net worth; card credentials not stored by merchants; merchant due diligence), 15 September 2025: https://rbi.org.in/Scripts/BS_ViewMasDirections.aspx?id=12896
- Read
Reserve Bank of India — Directions on authentication mechanisms for digital payment transactions (issued 25 September 2025, effective 1 April 2026; two factors; risk-based checks; SMS OTP not discontinued): https://rbi.org.in/Scripts/BS_PressReleaseDisplay.aspx?prid=61282
- Read
Business Today — RBI's debit, credit card rule to change from October 1 (card-on-file tokenisation from 1 October 2022), August 2022: https://www.businesstoday.in/latest/trends/story/rbis-debit-credit-card-rule-to-change-from-october-1-check-all-details-here-345354-2022-08-23
- Blog
SCC Online — UPI changes starting August 1 (NPCI's API guidelines of 21 May 2025: status checks only three times, 90 seconds apart), July 2025: https://www.scconline.com/blog/post/2025/07/30/upi-changes-starting-august-1-ncpi-guidelines-upi-api-usage-2025/
- Read
Business Today — UPI clocks 24.07 bn transactions worth ₹29.37 lakh crore in September (802 million a day), October 2026: https://www.businesstoday.in/latest/economy/story/upi-clocks-24-07-bn-transactions-worth-rs29-37-lakh-crore-in-september-as-festive-spending-gathers-pace-558981-2026-10-01
Design note: the architecture, code, thresholds, pricing, rates and dashboards in this case study are an illustrative design for the problem, not a description of Razorpay's internal systems beyond what Razorpay has published in the sources above.
Further Reading#
Related systems to study:
- Read
India's UPI (Bonus Problem 1 in this series): the switch on the other side of every UPI payment, with its own deemed-success state
- Read
Visa / Mastercard (Bonus Problem 4): the card network path, authorisation and settlement
- Read
BookMyShow (Bonus Problem 16) and Swiggy (Bonus Problem 17): merchants that depend on unknown outcomes being resolved correctly
Foundational concepts (this course):
- Read
Week 2, Day 2 — Idempotency in Practice
- Read
Week 2, Day 3 — Circuit Breakers
- Read
Week 2, Day 4 — Webhook Delivery
- Read
Week 3, Day 2 — Transactional Outbox
- Read
Week 3, Day 5 — Audit Log System
- Read
Week 5, Day 3 — Saga Orchestration
- Read
Week 9, Day 2 — Noisy Neighbour
"A billion payments a quarter, and somewhere every minute a bank that hasn't answered. The gateway that earns trust isn't the one that never meets silence. It's the one that never mistakes silence for an answer."
- 0/7stages cleared
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.