~learn.

Side quest 18Payment gateway

Razorpay Payment Gateway.

Every Rupee Exactly Once: Taking Payments for Millions of Businesses Through Banks That Sometimes Don't Answer

EXPERT39 MIN READ6 PHASES · 45 MIN INTERVIEW+120 XP

Mission brief

Design a payment gateway for online businesses

Quest log0/7 stages · 0/8 checkpoints · 0/8 quick checks
in annualised payments (Aug 2026)
$180 billion+
transactions a quarter
1 billion+
UPI payments in India in September 2026, 802 million a day
24.07 billion
for a merchant's server to answer a webhook, retried for 24 hours
5 s

Company names and logos are trademarks of their owners, shown only to identify each company. This course isn't affiliated with or endorsed by any of them.

Optional · timed run · 6 phases · 45 min

Want it harder? Hide the model answer and work each phase in a scratchpad against its time budget, then reveal and score yourself. Or simply read on: the checkpoints below work either way.

Intro · 3 min

How Do You Take a Payment Exactly Once When the Bank on the Other End Sometimes Never Replies?

It is 11:59:58 PM. An online store's festive sale starts at midnight. Forty thousand shoppers have full carts and their thumbs over "Pay". In two seconds, every one of them will press it.

Behind the button is a chain the shopper never sees: the store's server, the payment gateway, an acquiring bank or a UPI switch, a card network or NPCI, and the shopper's own bank, which has to ask for a PIN or an OTP. Any link can be slow. Some will not answer at all. And for every payment, the gateway must end up with one answer it can stand behind: the money moved, or it didn't.

If it guesses wrong, either a shopper is charged for an order the store never saw, or a store ships an order nobody paid for. At a billion transactions a quarter, "rarely" happens every minute.

Razorpay and Indian payments, from published numbers
  • $180 billion+ — Razorpay's annualised payment volume, and 1 billion+ transactions a quarter (its CEO, Business Today, August 2026)
  • 24.07 billion — UPI transactions in India in September 2026, worth ₹29.37 lakh crore, an average of 802 million a day (NPCI data)
  • 400 TPS — the peak, during the IPL, of the first version of Razorpay's UPI switch; the second version was designed for 10,000 TPS (Razorpay engineering, September 2026)
  • 70%+ — share of Razorpay's UPI volume that runs through that switch
  • T+2 working days — Razorpay's default settlement cycle for domestic payments; instant settlement is optional (Razorpay docs)
  • 3 days — an authorised payment not captured by then is refunded to the customer automatically (Razorpay docs)
  • 24 hours, 5 seconds — webhooks are retried with exponential backoff for 24 hours, and each delivery must be answered within 5 seconds (Razorpay docs)

For context:
Razorpay received the RBI's final authorisation as a payment aggregator in December 2023
The RBI's Master Direction on payment aggregators (15 September 2025) requires a non-bank aggregator to keep merchants' funds in an escrow account with a scheduled commercial bank
Since 1 October 2022, merchants can't store customers' card numbers; card-on-file payments use network tokens
Everything in this design that is not in the list above is an illustrative design target, not Razorpay's internal architecture

This is the system we'll design today.

ExplainerThe whole quest: Razorpay

Razorpay handles $180 billion+ a year and over a billion payments a quarter; designed for 10,000 TPS, each payment fans out into ~15 writes and events, and ~13,500 payments are unknown at any moment because NPCI allows only three UPI status checks 90 s apart. Idempotent APIs feed a payment state machine, a router picks processors by success rate per slice, state changes commit with ledger entries and an outbox, and a reconciler resolves unknowns. The deep dives cover idempotency, routing, unknown outcomes, the ledger and T+2 settlement and webhooks, then a sale midnight, one slow issuer and the daily three-way match.

The Interview Begins#

You're interviewing at a payments company. The principal engineer draws a line of boxes across the whiteboard: Store → Gateway → Bank → Network → Bank. Then she draws a question mark over the last arrow.

Interviewer

"Every payments interview ends up at this question mark. A business integrates with us once and wants to accept every method India has. Design the gateway: taking the payment, routing it, knowing whether it worked, telling the merchant, and paying them. Then tell me what you do when the question mark doesn't answer."

Design a payment gateway for online businesses

Businesses create orders through an API and send customers to a checkout. Customers pay by card, UPI, net banking or wallet. The gateway sends each payment to one of several banks and processors, learns the result, informs the merchant, and settles the money to the merchant's bank account.

Requirements
  • A customer is charged at most once per order, however many times anyone retries
  • Every payment ends in exactly one final state: captured and settled, or failed or refunded
  • Route each payment where it is most likely to succeed; survive any one bank or processor failing
  • Tell the merchant about every state change, reliably, even when their server is down
  • Pay merchants the right amount on the agreed schedule, from funds held in escrow
  • Never store card numbers outside a secured vault; follow RBI and PCI-DSS rules
Interviewer

"I care about one word: exactly. Show me where 'exactly once' is real, and where you build it out of 'at least once' plus something clever."

Phase 1 · Stage 1 of 7 · 5 min budget · 3 min read

Phase 1: Requirements Clarification (5 minutes)#

You

"Before boxes, I want to know which parts we control, and how slow the parts we don't control can be."

Your Questions#

Transcript
You

"Do we hold the money, or does it go straight from the customer's bank to the merchant's?"

Interviewer

"As a payment aggregator, the money lands in an escrow account at a bank, held for the merchants, and we instruct payouts from it. The RBI's 2025 Master Direction requires exactly that for non-bank aggregators. Razorpay's default settlement is T+2 working days, with instant settlement as an option."

You

"Which methods, and which ones are asynchronous?"

Interviewer

"Cards, UPI, net banking, wallets. UPI is asynchronous by nature: you send a request, the customer approves it in their UPI app, and the result comes back later as a callback. Razorpay's engineers describe a message bus at the heart of their UPI switch for exactly that reason. Cards usually need an authentication step at the customer's bank, which can be slow."

You

"What's the volume?"

Interviewer

"Razorpay says it handles more than a billion transactions a quarter and more than $180 billion a year. The first version of its UPI switch peaked at 400 TPS during the IPL; the next was built for 10,000. Size it."

You

"When a bank doesn't answer, can we just ask it again?"

Interviewer

"Within limits. For UPI, NPCI's rules since August 2025 allow a transaction status check only three times, at least 90 seconds apart. So you can't poll your way out of uncertainty."

You

"What does the merchant integrate with?"

Interviewer

"An API to create orders and fetch payments, a checkout the customer sees, and webhooks to the merchant's server for every state change. Payments move through created, authorised and captured; an authorised payment that isn't captured within 3 days is refunded automatically."

You

"And card data?"

Interviewer

"Since October 2022, merchants can't store card numbers; saved cards use network tokens. The card data you do touch lives in a PCI-DSS vault, and the RBI's authentication rules from April 2026 require two factors, one of them dynamic, with issuers free to add risk-based checks."

Functional Requirements#

Functional requirements
1. Merchant API
  • Create an order (amount, currency, receipt); fetch orders, payments, refunds
  • Every write API takes an idempotency key
  • Capture an authorised payment (automatically or by API); refund fully or partly
2. Checkout
  • Show methods by merchant configuration and device; tokenised saved cards
  • Collect card details into the vault, or start UPI intent / QR / collect, or redirect to net banking
3. Processing
  • Route each payment to a processor (acquiring bank, UPI switch, net-banking connection) by method, bank and live success rates
  • Run authentication (card OTP or other factors, UPI PIN in the customer's app)
  • Track every payment to a final state; resolve unknowns by status checks and settlement files
4. Money
  • Double-entry ledger of every movement: captured, fees, taxes, refunds, chargebacks, settlements
  • Settle to merchants on schedule (T+2 by default, instant on request) from escrow
5. Notifications
  • Webhooks for every state change, signed, retried, with event ids for deduplication
  • Dashboard and reports for merchants

Non-Functional Requirements#

Non-functional requirements
Scale
  • ~1 billion+ transactions a quarter; design for a 10,000 TPS peak (design target, the same order as the UPI switch's own target)
Correctness
  • At most one capture per order; no payment left in a non-final state beyond its reconciliation window
  • The ledger always balances; settlements equal captured minus fees, taxes and refunds
Latency
  • Gateway's own work (create, route, persist) < 100 ms p99, excluding the bank and the customer
  • Payment result to the merchant's server: webhook sent < 5 s after we learn it
Availability
  • 99.99% for the payment path (design target); any one processor or bank can fail without taking the gateway down
  • One merchant's spike can't starve other merchants
Security and compliance
  • PCI-DSS card vault; no card numbers outside it; tokens for saved cards
  • Funds in escrow; merchant due diligence before onboarding; audit trail for every money movement

Phase 2 · Stage 2 of 7 · 5 min budget · 2 min read

Phase 2: Back of the Envelope Estimation (5 minutes)#

You

"The transaction rate is modest by internet standards. What makes payments heavy is the number of things that happen per payment, and the number of payments whose outcome we don't know yet."

Your turn · checkpoint 1not answered

Before the numbers: how many unknowns?

At the 10,000 TPS design peak, suppose 1% of UPI payments (say half of all traffic) haven't got a definite answer 30 seconds after the customer approved. NPCI allows a status check only three times, 90 seconds apart. How many payments are in "unknown" at any moment, and what does that tell you?

Think it through, or jot it down. Then compare.

One strong answer

  • New unknowns: 10,000 × 50% × 1% = 50 a second.
  • How long they stay unknown: if the first status check is at 90 s and the last at 270 s, an unresolved payment can stay unknown for ~4.5 minutes, longer if only the settlement file settles it.
  • How many at once: 50 × 270 s ≈ 13,500 payments in "unknown" at the peak, plus a tail waiting for the next day's files.

So "unknown" isn't an error path; it's a normal state with its own storage, timers and reconciliation. The design must keep the customer and the merchant informed while those 13,500 resolve, and must never let a retry turn one of them into two charges.

Traffic Estimation#

WorksheetPayments and the work behind them (design target, rounded from public figures)
  • Public baseline: 1 billion+ transactions a quarter = ~11 million a day, ~127 a second on average
  • Peaks: sale midnights, festive evenings and match nights, assumed ~10× average for the busiest minutes: ~1,300 TPS
  • Design target with headroom: 10,000 TPS
  • Average transaction: $180 billion ÷ ~4 billion a year ≈ $45
  • Internal work per payment (order, payment, routing decision, processor call, callback, capture, ~4 ledger entries, ~3 webhooks, notifications): ~15 writes or events
  • At the design peak: 10,000 × 15 = ~150,000 internal operations a second
  • Webhooks: ~3 per payment → ~30,000 a second at the design peak, each with up to 24 hours of retries if a merchant's server is down

Storage Estimation#

WorksheetMoney data is small per row and kept for years
  • Payment records: 11 million a day × ~2 KB (amount, method, token reference, processor references, state history, risk score) = ~22 GB a day
  • Ledger entries: ~4 per payment × ~200 bytes = ~9 GB a day
  • Webhook delivery log: ~3 per payment × ~1 KB = ~33 GB a day, kept hot for weeks
  • Retention for money records (design assumption): 10 years → payments and ledger ≈ 31 GB × 365 × 10 ≈ ~113 TB
  • Card vault: tokens and encrypted card data in a separate, small, heavily isolated store

Key Metrics Summary#

Metric Value
Average rate ~127 TPS
Busiest minutes (assumed 10×) ~1,300 TPS
Design peak 10,000 TPS
Internal operations at the design peak ~150,000/s
Webhooks at the design peak ~30,000/s
Payments in "unknown" at the peak ~13,500
Payments + ledger, 10 years ~113 TB
Quick check3 questions · +5 XP each

Requirements and estimation

  1. Q1/3 1 billion transactions a quarter is roughly how many a second on average?

    Answer & explanation

    Answer: B

    1,000,000,000 ÷ 91 days ÷ 86,400 s ≈ 127 a second. The design peak of 10,000 TPS leaves room for sale midnights and growth.

  2. Q2/3 Why does the estimate count ~15 internal operations per payment?

    Answer & explanation

    Answer: C

    The capacity that matters is internal: at 10,000 TPS the gateway does ~150,000 writes and events a second. Sizing only for "payments" undersizes the system 15×.

  3. Q3/3 About 13,500 payments sit in "unknown" at the design peak. What does that mean for the design?

    Answer & explanation

    Answer: C

    NPCI allows only three status checks, 90 s apart, so uncertainty lasts minutes. Treating it as a first-class state is what keeps retries from becoming double charges.

Phase 3 · Stage 3 of 7 · 10 min budget · 3 min read

Phase 3: High-Level Design (10 minutes)#

Your turn · checkpoint 2not answered

Sketch it first: one payment, from "Pay" to settlement

Trace one ₹1,999 card payment from the merchant creating an order to the money arriving in the merchant's bank. Which steps must be exactly once, which are at least once with deduplication, and which can run hours later?

Think it through, or jot it down. Then compare.

One strong answer

  • Exactly once (by construction): creating the order and the payment, keyed on idempotency keys, and the capture, which is a state transition on one row guarded by its current state.
  • At least once, deduplicated: processor callbacks, status checks, webhooks to the merchant (with event ids), retries from the merchant's server. Each is safe to repeat because the receiving side checks state or ids.
  • The payment path: checkout → vault (card to token) → router (pick a processor by live success rates) → processor and bank, with authentication → result → state machine → ledger entries → outbox → webhook.
  • Hours later: reconciliation against processors' settlement files (T+1), and settlement to the merchant from escrow (T+2 by default), both driven by the ledger.
You

"I'll structure it around one rule: the payment state machine and the ledger are the only truth. Everything else, checkout, routing, processors, webhooks, either feeds them or reads from them. Processors are untrusted for timing, so their answers arrive as events, and anything we don't hear about gets reconciled."

System Architecture#

DiagramPayment gateway architecture
%% title: Payment gateway architecture
flowchart TD
  subgraph L1["Layer 1 · Merchants and customers"]
    direction LR
    MS@{ icon: "sd:server", label: "Merchant server<br/>orders, capture, refunds" }
    CK@{ icon: "sd:phone", label: "Checkout<br/>web / app SDK" }
  end
  GW@{ icon: "sd:gateway", label: "API gateway<br/>auth, idempotency, rate limit per merchant" }
  subgraph L3["Layer 3 · Payment core"]
    direction LR
    PAY["Payment service<br/>state machine, one row per payment"]:::hot
    RT["Router<br/>success rates per method, bank, processor"]:::core
    RISK["Risk checks<br/>velocity, device, amount"]:::service
  end
  subgraph L4["Layer 4 · Processors"]
    direction LR
    VAULT@{ icon: "sd:lock", label: "Card vault<br/>PCI-DSS, tokens" }
    ADP["Processor adapters<br/>acquirers, UPI switch, net banking"]:::service
    BANKS@{ icon: "sd:bank", label: "Banks, networks, NPCI" }
  end
  subgraph L5["Layer 5 · Money and events"]
    direction LR
    LED@{ icon: "sd:database", label: "Ledger<br/>double-entry, append-only" }
    BUS@{ icon: "tech:kafka", label: "Kafka<br/>payment events (outbox)" }
    REC["Reconciler<br/>status checks, settlement files"]:::core
  end
  subgraph L6["Layer 6 · Out to merchants"]
    direction LR
    WH@{ icon: "sd:webhook", label: "Webhook delivery<br/>queue per merchant, retries 24 h" }
    SET["Settlement<br/>escrow → merchant, T+2"]:::service
    ESC@{ icon: "sd:bank", label: "Escrow bank" }
  end
  MS ==> GW
  CK ==> GW
  GW ==> PAY
  PAY --> RISK
  PAY ==> RT
  RT ==> ADP
  CK -.->|"card details"| VAULT
  ADP ==> BANKS
  PAY ==> LED
  PAY -.-> BUS
  BUS --> WH
  BUS --> REC
  LED --> SET --> ESC
  class MS,CK client
  class GW edge
  class VAULT edge
  class BANKS,ESC external
  class LED store
  class BUS queue
  class WH queue

KEY INSIGHTS:

  • The payment service owns one row per payment and is the only thing that moves it between states. Processor callbacks, status checks and merchant calls all become state-transition requests it can accept or refuse
  • The card vault is a separate, minimal system: checkout sends card details straight to it and gets a token; the rest of the gateway never sees a card number
  • Every state change and its ledger entries commit together, and leave through an outbox onto Kafka (Razorpay's engineers describe the same outbox pattern in their UPI switch); webhooks and reconciliation are consumers

Data Flow: One Card Payment#

You

"Let's follow Meera paying ₹1,999 at a store that auto-captures."

DiagramOne card payment, order to webhook
%% title: One card payment, order to webhook
sequenceDiagram
  autonumber
  participant M as Merchant server
  participant G as Gateway
  participant V as Vault
  participant R as Router
  participant B as Acquirer + issuer
  participant W as Webhooks
  M->>G: Create order ₹1,999 (idem key o-77)
  G-->>M: order_A1
  Note over G: Meera opens checkout, enters card
  G->>V: Card details
  V-->>G: token tk_9
  G->>R: Route card, issuer bank X
  R-->>G: Acquirer P2 (best success rate now)
  G->>B: Authorise tk_9 ₹1,999
  B-->>G: Needs OTP, then authorised
  Note over G: payment pay_B1: CREATED → AUTHORIZED → CAPTURED · ledger entries in the same commit
  G-)W: payment.captured, order.paid (event ids)
  W->>M: POST webhook, signed
  M-->>W: 200 in 120 ms

MEANWHILE, IN PARALLEL:

  • The router's success-rate window for "card, issuer X, acquirer P2" just got one more success
  • The reconciler will see pay_B1 again tomorrow, in P2's settlement file, and match it against the ledger
  • On day T+2, pay_B1's net amount is part of the store's settlement from escrow
Quick check3 questions · +5 XP each

High-level design

  1. Q1/3 Why does checkout send card details straight to the vault instead of through the payment service?

    Answer & explanation

    Answer: B

    The rest of the gateway works with tokens. A breach or a log line elsewhere can then never leak card numbers, and only the vault carries the heaviest compliance burden.

  2. Q2/3 A processor's callback says "success" for a payment the gateway already marked failed after a timeout. Who decides what happens?

    Answer & explanation

    Answer: C

    Only the payment service moves a payment between states. A late success on a failed payment becomes a defined path (capture if the order still wants it, otherwise refund), never an overwrite.

  3. Q3/3 Why do webhooks and reconciliation consume events from Kafka rather than being called directly by the payment service?

    Answer & explanation

    Answer: C

    If the payment service called each consumer, a slow merchant or a crashed reconciler would block or lose events. The outbox plus consumers turns that into durable, independent work.

Phase 4 · Stage 4 of 7 · 20 min budget · 17 min read

Phase 4: Deep Dives (20 minutes)#

Deep Dive 1: Idempotency and the Payment State Machine (Week 2 Concepts: Idempotency, Week 5: Consistency Models)#

You

"'Exactly once' in payments is two promises: each request has one effect, however often it's sent, and each payment has one final state, however many messages arrive about it."

Your turn · checkpoint 3not answered

The merchant retried

A merchant's server sends "create order ₹1,999" and its connection drops before the reply. It retries with the same idempotency key. Meanwhile, a different bug in the merchant's code sends the same key with ₹2,999. And on our side, a processor's success callback arrives twice. What happens in each case?

Think it through, or jot it down. Then compare.

One strong answer

  • Same key, same body: the first request is stored with its key and its response; the retry returns that stored response. One order.
  • Same key, different body: the key is bound to a hash of the request body; a mismatch is rejected with an error. Silently returning the ₹1,999 order would hide a merchant bug; creating a second order would break the key's meaning.
  • A retry while the first is still running: the key is locked while in flight; the retry waits briefly or gets "in progress, retry later", never a second execution.
  • Duplicate callback: the transition is guarded by the current state: AUTHORIZED → CAPTURED only applies if the row is in AUTHORIZED. The second callback finds it CAPTURED and is acknowledged without effect.

The Challenge:

Where duplicates come from
  • Merchants retry on timeouts, and their HTTP clients retry automatically
  • Customers tap "Pay" twice, or reopen a payment page from history
  • Processors resend callbacks; UPI results can arrive as a callback and as a status-check answer
  • Our own workers crash between doing something and recording that they did it

The Solution:

DiagramPayment state machine
%% title: Payment state machine
stateDiagram-v2
  [*] --> CREATED
  CREATED --> PENDING: sent to processor
  PENDING --> AUTHORIZED: bank approved
  PENDING --> FAILED: bank declined
  PENDING --> UNKNOWN: no answer in time
  UNKNOWN --> AUTHORIZED: status check or callback says yes
  UNKNOWN --> FAILED: says no, or settlement file has no record
  AUTHORIZED --> CAPTURED: merchant or auto-capture
  AUTHORIZED --> AUTO_REFUNDED: not captured in 3 days
  CAPTURED --> REFUNDED: merchant refunds
  FAILED --> LATE_SUCCESS: money moved after all
  LATE_SUCCESS --> CAPTURED: order still open
  LATE_SUCCESS --> AUTO_REFUNDED: order closed
  CREATED: CREATED · idempotency key stored
  UNKNOWN: UNKNOWN · reconcile, max 3 UPI checks
  AUTHORIZED: AUTHORIZED · funds blocked
  CAPTURED: CAPTURED · ledger entries posted
  class CREATED,PENDING,AUTHORIZED core
  class CAPTURED ok
  class FAILED,REFUNDED,AUTO_REFUNDED bad
  class UNKNOWN,LATE_SUCCESS queue

KEY DESIGN DECISIONS:

  1. AN IDEMPOTENCY KEY IS A RECORD, NOT A CACHE ENTRY — Key, merchant, request hash, status (in progress / done) and the stored response, in the same database as the effect, so "done" and "the order exists" commit together.
  2. TRANSITIONS ARE CONDITIONAL UPDATES — UPDATE payments SET state = 'CAPTURED' WHERE id = ? AND state = 'AUTHORIZED'. Row count 1 means this request did it; 0 means someone else already did.
  3. UNKNOWN IS A STATE — A timeout never becomes "failed" by itself. It becomes UNKNOWN with a reconciliation schedule, and the customer sees "we're confirming your payment".
  4. LATE SUCCESS HAS A PATH — If the money moved after we called a payment failed, it goes to LATE_SUCCESS and then to capture or automatic refund, depending on whether the merchant's order is still open.
  5. AUTHORISED IS NOT FOREVER — Following the documented rule, an authorised payment that isn't captured within 3 days is refunded automatically.
pythonStored · 71 lines
# api/idempotency.py

"""
Idempotent write APIs. The key, a hash of the request, and the stored
response live in the same database as the effect, written in the same
transaction, so a crash can never leave "effect done, key not recorded".
"""

import hashlib
import json
from dataclasses import dataclass
from typing import Callable, Optional


@dataclass
class Stored:
    status: str              # IN_PROGRESS | DONE
    request_hash: str
    response: Optional[dict]


class IdempotencyMismatch(Exception):
    pass


class InProgress(Exception):
    pass


def request_hash(method: str, path: str, body: dict) -> str:
    return hashlib.sha256(f"{method} {path} {json.dumps(body, sort_keys=True)}".encode()).hexdigest()


class IdempotentHandler:
    def __init__(self, db):
        self.db = db

    def run(self, merchant_id: str, key: str, method: str, path: str, body: dict,
            effect: Callable[[object], dict]) -> dict:
        h = request_hash(method, path, body)
        with self.db.transaction() as tx:
            inserted = tx.execute(
                "INSERT INTO idem_keys (merchant_id, key, request_hash, status) "
                "VALUES (%s, %s, %s, 'IN_PROGRESS') ON CONFLICT DO NOTHING",
                (merchant_id, key, h),
            ).rowcount
            if inserted == 0:
                row: Stored = tx.fetch_one(
                    "SELECT status, request_hash, response FROM idem_keys "
                    "WHERE merchant_id = %s AND key = %s FOR UPDATE", (merchant_id, key))
                if row.request_hash != h:
                    raise IdempotencyMismatch(key)            # same key, different request
                if row.status == "DONE":
                    return row.response                       # a retry: same answer
                raise InProgress(key)                         # first attempt still running

            response = effect(tx)                             # creates the order, in this tx
            tx.execute(
                "UPDATE idem_keys SET status = 'DONE', response = %s "
                "WHERE merchant_id = %s AND key = %s",
                (json.dumps(response), merchant_id, key),
            )
            return response


def transition(tx, payment_id: str, frm: str, to: str) -> bool:
    """The only way a payment changes state. False means someone else got there first."""
    return tx.execute(
        "UPDATE payments SET state = %s, updated_at = now() WHERE id = %s AND state = %s",
        (to, payment_id, frm),
    ).rowcount == 1
You

"The effect inside run only touches our own database. Calls to processors are never made inside that transaction: the payment is created first, then the processor call happens, and its result comes back as a transition. That's how a crash halfway through leaves a payment in PENDING, which reconciliation knows how to finish, instead of a charge nobody recorded."

Deep Dive 2: Routing to the Processor Most Likely to Say Yes (Week 2 Concepts: Circuit Breakers, Week 10: Observability)#

You

"The same card payment can often go through several acquiring banks. On a good day they're similar; on a bad day one of them, or one issuing bank behind it, is failing half its payments. Routing is where the gateway turns that into a higher success rate for merchants."

Your turn · checkpoint 4not answered

One route is failing

For Visa cards from issuer X, acquirer P1 normally approves 92% and P2 89%. In the last five minutes, P1's success rate for that slice dropped to 60%, but only for issuer X. What should the router do, how quickly, and how does it know when P1 has recovered?

Think it through, or jot it down. Then compare.

One strong answer

  • Measure per slice, not per processor: keep success rates per (method, network, issuer, processor) over a short sliding window, because failures are usually local: one issuer through one acquirer.
  • Move traffic fast: when a slice drops well below its normal rate, with enough attempts to be sure, mark that route down for that slice and send its traffic to the next-best processor. Razorpay's documentation for its Optimizer product describes this behaviour: a temporary downtime of 20 minutes when a high-priority gateway's success rate drops, with traffic moving to the next priority.
  • Keep probing: while a route is down, send a small share of traffic (say 2%) to it to learn when it recovers, and bring it back gradually.
  • Separate customer failures from route failures: a wrong OTP or insufficient funds is the customer's answer, not the processor's fault; count only technical failures and bank-side declines that differ between routes.
  • Respect merchant rules: some merchants pin processors for contract or pricing reasons; route within what they allow.

The Challenge:

Why a static route fails
  • An issuer's authentication page slows down, and every payment sent there through one acquirer times out
  • A processor's overall success rate looks fine while one bank behind it fails half the time
  • Switching on noise sends traffic back and forth every minute
  • A route that is never tried again after an outage is never trusted again

The Solution:

DiagramThe routing decision for one payment
%% title: The routing decision for one payment
flowchart TD
  P["Payment<br/>card · Visa · issuer X · ₹1,999"]:::core
  ALLOW["Merchant's allowed processors<br/>and priorities"]:::service
  SR@{ icon: "sd:chart", label: "Success rate per slice<br/>last 5 min + 1 h baseline" }
  DOWN{"Route marked down<br/>for this slice?"}:::cache
  PICK["Pick best score<br/>success rate, cost, latency"]:::hot
  PROBE["2% probe traffic<br/>to down routes"]:::service
  ADP["Processor adapter"]:::service
  P --> ALLOW --> DOWN
  SR --> DOWN
  DOWN -->|"no"| PICK
  DOWN -->|"yes, probe share"| PROBE
  PICK ==> ADP
  PROBE --> ADP
  ADP -.->|"result"| SR
  class SR cache
pythonSliceStats · 63 lines
# routing/router.py

"""
Route a payment to the processor most likely to approve it, using
success rates per slice (method, network, issuer, processor). A slice
that drops far below its baseline is marked down for a while; a small
probe share keeps testing it so it can come back.
"""

import random
from collections import deque
from dataclasses import dataclass, field
from typing import Deque, Dict, List, Optional, Tuple

WINDOW_S = 300
MIN_ATTEMPTS = 50
DOWN_FOR_S = 20 * 60           # the documented Optimizer behaviour uses 20 minutes
PROBE_SHARE = 0.02


@dataclass
class SliceStats:
    events: Deque[Tuple[float, bool]] = field(default_factory=deque)   # (ts, technical_ok)
    baseline: float = 0.9
    down_until: float = 0.0

    def rate(self, now: float) -> Optional[float]:
        while self.events and self.events[0][0] < now - WINDOW_S:
            self.events.popleft()
        if len(self.events) < MIN_ATTEMPTS:
            return None
        return sum(ok for _, ok in self.events) / len(self.events)


class Router:
    def __init__(self, rng: random.Random):
        self.stats: Dict[Tuple[str, str], SliceStats] = {}
        self.rng = rng

    def record(self, slice_key: str, processor: str, technical_ok: bool, now: float) -> None:
        """Called with each result. Customer declines (wrong OTP, no funds) are not recorded."""
        s = self.stats.setdefault((slice_key, processor), SliceStats())
        s.events.append((now, technical_ok))
        r = s.rate(now)
        if r is not None and r < s.baseline - 0.15:
            s.down_until = now + DOWN_FOR_S                      # move traffic away, for a while

    def choose(self, slice_key: str, allowed: List[Tuple[str, int]], now: float) -> str:
        """allowed: (processor, merchant priority), lower priority number first."""
        up, down = [], []
        for proc, prio in sorted(allowed, key=lambda a: a[1]):
            s = self.stats.get((slice_key, proc))
            (down if s and s.down_until > now else up).append((proc, prio, s))
        if down and (not up or self.rng.random() < PROBE_SHARE):
            return down[0][0]                                     # keep learning about down routes
        if not up:
            return sorted(allowed, key=lambda a: a[1])[0][0]

        def score(item):
            proc, prio, s = item
            r = (s.rate(now) if s else None) or (s.baseline if s else 0.9)
            return r - prio * 0.01                                # merchant priority breaks near-ties
        return max(up, key=score)[0]
Quick check4 questions · +5 XP each

Idempotency and routing

  1. Q1/4 A merchant sends the same idempotency key twice, the second time with a different amount. What should the API do?

    Answer & explanation

    Answer: C

    A key means "this exact request". A different body under the same key is a bug on the merchant's side, and both silent options would hide it.

  2. Q2/4 Two success callbacks arrive for the same payment. Why is the second one harmless?

    Answer & explanation

    Answer: B

    UPDATE ... WHERE state = 'AUTHORIZED' affects one row the first time and none the second. The duplicate is acknowledged with no effect.

  3. Q3/4 Acquirer P1's success rate drops to 60% for issuer X only. Why keep stats per (method, network, issuer, processor) rather than per processor?

    Answer & explanation

    Answer: B

    A per-processor average would barely move, or would wrongly move all traffic off P1. Slicing finds the failing route and moves only that traffic.

  4. Q4/4 Why send ~2% of traffic to a route that is marked down?

    Answer & explanation

    Answer: C

    Without probes, a route that failed once is never measured again. A small share tells the router when the success rate is back to its baseline.

Deep Dive 3: The Bank Didn't Answer, Reconciling Unknown Outcomes (Week 5 Concepts: Saga Orchestration, Week 3: Audit Logs)#

You

"Every payment system ends up here. The customer approved the payment in their UPI app, and we never heard back. The money may have moved, or not. We must find out, without asking too often and without ever charging twice."

Your turn · checkpoint 5not answered

The UPI result never came

Meera approves a UPI payment. Thirty seconds later there's no callback. NPCI allows three status checks, at least 90 seconds apart. What do you show Meera and the merchant now, what do you do over the next few minutes, and what's your final safety net if the checks don't settle it?

Think it through, or jot it down. Then compare.

One strong answer

  • Now: the payment goes to UNKNOWN. Meera sees "we're confirming your payment, don't pay again"; the merchant's order stays "attempted", not "failed", and the merchant is told not to ship and not to ask for a new payment yet.
  • Next few minutes: a callback may still arrive and settles it. If not, status checks at about 90 s, 180 s and 270 s after the payment, each keyed on the same transaction reference. A definite answer moves the payment to AUTHORIZED (then capture) or FAILED.
  • If still unknown: wait for the processor's settlement or MIS file (T+1). A payment in the file is a success; a payment missing from it is a failure.
  • If money moved after we said "failed": the payment goes to LATE_SUCCESS: captured if the merchant's order is still open, otherwise refunded automatically, and both sides are told.
  • Never: create a new payment for the same order while one is UNKNOWN, or turn a timeout into "failed" by itself.

The Challenge:

Why you can't just ask again
  • NPCI allows a UPI status check only three times, 90 seconds apart, since August 2025
  • Banks answer status checks from the same systems that are slow right now
  • Customers retry with a new payment when they see "failed", so a premature "failed" creates double charges
  • Some answers only exist in the next day's settlement files

The Solution:

DiagramResolving an unknown UPI payment
%% title: Resolving an unknown UPI payment
sequenceDiagram
  autonumber
  participant G as Payment service
  participant S as UPI switch / processor
  participant R as Reconciler
  participant F as Settlement files
  G->>S: Collect / intent request (txn ref T1)
  Note over G: 30 s, no callback · state UNKNOWN · customer told "confirming"
  R->>S: Status check 1 at 90 s (T1)
  S-->>R: Pending
  R->>S: Status check 2 at 180 s (T1)
  alt definite answer
    S-->>R: Success
    R->>G: UNKNOWN → AUTHORIZED, then capture
  else still pending after check 3
    R->>F: Next day: is T1 in the settlement file?
    F-->>R: T1 present
    R->>G: Late success: capture if order open, else refund
  end
pythonPayment · 84 lines
# recon/reconciler.py

"""
Drives every UNKNOWN payment to a final state. Callbacks are handled
elsewhere; this worker covers the payments we heard nothing about,
within processors' limits on status checks, and falls back to
settlement files. Every action is a guarded transition.
"""

from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional

UPI_CHECK_AT = [timedelta(seconds=90), timedelta(seconds=180), timedelta(seconds=270)]


@dataclass
class Payment:
    id: str
    order_id: str
    method: str
    processor: str
    processor_ref: str
    state: str
    created_at: datetime
    checks_done: int = 0


class Reconciler:
    def __init__(self, db, adapters, orders, scheduler, notifier):
        self.db = db
        self.adapters = adapters
        self.orders = orders
        self.scheduler = scheduler
        self.notifier = notifier

    def on_unknown(self, p: Payment) -> None:
        for delay in UPI_CHECK_AT if p.method == "upi" else [timedelta(seconds=30), timedelta(minutes=5)]:
            self.scheduler.at(p.created_at + delay, "status_check", p.id)

    def status_check(self, payment_id: str) -> None:
        p = self.db.get_payment(payment_id)
        if p.state != "UNKNOWN":
            return                                     # a callback already settled it
        answer = self.adapters[p.processor].status(p.processor_ref)   # same reference every time
        self.db.increment_checks(p.id)
        if answer == "SUCCESS":
            self._settle(p, success=True)
        elif answer == "FAILED":
            self._settle(p, success=False)
        # "PENDING": wait for the next scheduled check, or the settlement file

    def on_settlement_file_row(self, processor_ref: str, amount_paise: int) -> None:
        p = self.db.get_payment_by_ref(processor_ref)
        if p.state == "UNKNOWN":
            self._settle(p, success=True)
        elif p.state == "FAILED":
            self._late_success(p)                      # money moved after we said no
        # CAPTURED / REFUNDED: matched, nothing to do except mark reconciled

    def close_unmatched(self, file_date: datetime) -> None:
        """After the file for a day is complete, UNKNOWN payments not in it failed."""
        for p in self.db.unknown_created_before(file_date):
            self._settle(p, success=False)

    def _settle(self, p: Payment, success: bool) -> None:
        with self.db.transaction() as tx:
            if success:
                if self.db.transition(tx, p.id, "UNKNOWN", "AUTHORIZED"):
                    self.db.outbox(tx, "payment.authorized", p.id)
            elif self.db.transition(tx, p.id, "UNKNOWN", "FAILED"):
                self.db.outbox(tx, "payment.failed", p.id)

    def _late_success(self, p: Payment) -> None:
        with self.db.transaction() as tx:
            if not self.db.transition(tx, p.id, "FAILED", "LATE_SUCCESS"):
                return
            if self.orders.is_open(p.order_id, tx=tx):
                self.db.transition(tx, p.id, "LATE_SUCCESS", "CAPTURED")
                self.db.outbox(tx, "payment.captured", p.id)
            else:
                self.db.transition(tx, p.id, "LATE_SUCCESS", "AUTO_REFUNDED")
                self.db.outbox(tx, "refund.created", p.id)
        self.notifier.customer_and_merchant(p.id)
You

"The reconciler also runs the other way every day: for every row in every processor's settlement file, there must be a ledger entry with the same reference and amount, and for every captured payment, a file row. Anything unmatched is a ticket for a person, not something the code guesses about."

Deep Dive 4: The Ledger, Escrow and Settlement (Week 3 Concepts: Audit Log System, Week 5: Consistency)#

You

"Payment states say what happened to a payment. The ledger says where every rupee is. They are different questions, and the second one is what auditors, regulators and merchants care about."

Your turn · checkpoint 6not answered

Where is the money right now?

Meera's ₹1,999 is captured at 10:00 on Monday. The gateway's fee is 2% plus 18% GST on the fee. On Tuesday the merchant refunds ₹500. Settlement is T+2 working days. Which accounts move, by how much, and how do you prove nothing was lost?

Think it through, or jot it down. Then compare.

One strong answer

  • On capture (Monday): the money is owed to us by the processor until it reaches escrow. Entries: debit "processor receivable" ₹1,999; credit "merchant payable" ₹1,999. Then the fee: debit "merchant payable" ₹47.18; credit "fee revenue" ₹39.98 and "GST payable" ₹7.20.
  • When the processor's funds reach escrow: debit "escrow cash" ₹1,999; credit "processor receivable" ₹1,999.
  • On refund (Tuesday): debit "merchant payable" ₹500; credit "refund payable to customer" ₹500, then cleared when the refund goes out.
  • On settlement (Wednesday, T+2): pay the merchant what is left: ₹1,999 − ₹47.18 − ₹500 = ₹1,451.82. Debit "merchant payable"; credit "escrow cash".
  • Proof: every posting is a set of entries that sums to zero; the ledger is append-only; the escrow account's bank statement must equal the "escrow cash" balance every day. A difference is an incident, not a rounding error.

The Challenge:

What a balance column gets wrong
  • Updating a merchant's "balance" in place loses why it changed, and two concurrent updates lose one of them
  • Refunds, chargebacks and fees arrive on different days from the payment
  • The bank, the processor and our records each have their own view of the same money
  • An aggregator must keep merchants' funds in escrow, so our numbers must match a bank statement every day

The Solution:

DiagramMoney from customer to merchant
%% title: Money from customer to merchant
flowchart TD
  CUS@{ icon: "sd:user", label: "Customer's bank" }
  PRC@{ icon: "sd:bank", label: "Processor / acquirer" }
  ESC@{ icon: "sd:bank", label: "Escrow account<br/>scheduled commercial bank" }
  LED["Ledger<br/>double-entry, append-only"]:::hot
  SET["Settlement run<br/>T+2, or instant"]:::core
  MER@{ icon: "sd:building", label: "Merchant's bank account" }
  CUS ==>|"₹1,999"| PRC
  PRC ==>|"funds, T+1"| ESC
  ESC ==>|"₹1,451.82"| MER
  LED -.->|"payable per merchant"| SET
  SET ==> ESC
  class CUS client
  class PRC,MER external
  class ESC store
pythonEntry · 57 lines
# ledger/postings.py

"""
Double-entry ledger. Every business event is one posting: a set of
entries that sums to zero, written in one transaction and never
updated. Balances are sums of entries (cached, but always rebuildable).
Amounts are integer paise; never floats.
"""

from dataclasses import dataclass
from typing import List

FEE_BPS = 200            # 2% (illustrative pricing)
GST_BPS = 1800           # 18% on the fee


@dataclass(frozen=True)
class Entry:
    account: str         # e.g. "merchant_payable:m_42"
    amount_paise: int    # debit positive, credit negative


class UnbalancedPosting(Exception):
    pass


class Ledger:
    def __init__(self, db):
        self.db = db

    def post(self, tx, posting_id: str, kind: str, entries: List[Entry]) -> None:
        if sum(e.amount_paise for e in entries) != 0:
            raise UnbalancedPosting(posting_id)
        # posting_id is unique: replaying the same event can never post twice
        tx.execute("INSERT INTO postings (id, kind) VALUES (%s, %s)", (posting_id, kind))
        for e in entries:
            tx.execute("INSERT INTO entries (posting_id, account, amount_paise) VALUES (%s, %s, %s)",
                       (posting_id, e.account, e.amount_paise))


def capture_entries(merchant: str, processor: str, amount: int) -> List[Entry]:
    fee = amount * FEE_BPS // 10_000
    gst = (fee * GST_BPS + 5_000) // 10_000      # rounded to the nearest paisa
    return [
        Entry(f"processor_receivable:{processor}", amount),
        Entry(f"merchant_payable:{merchant}", -amount),
        Entry(f"merchant_payable:{merchant}", fee + gst),
        Entry("fee_revenue", -fee),
        Entry("gst_payable", -gst),
    ]


def settlement_entries(merchant: str, payable: int) -> List[Entry]:
    return [
        Entry(f"merchant_payable:{merchant}", payable),
        Entry("escrow_cash", -payable),
    ]

A few details carry the correctness:

  • The capture's state transition and its posting commit in the same transaction as the outbox event, so a payment is never CAPTURED without its entries.
  • Postings are keyed (capture:pay_B1, refund:rf_3): a replayed event inserts nothing, because the posting id already exists.
  • Settlement reads balances, not payments: the amount to pay a merchant is the balance of merchant_payable:m_42 for entries older than the cut-off, which already includes fees, refunds and chargebacks.
  • Every day, three numbers must agree: the ledger's escrow_cash, the escrow bank's statement, and the sum of processors' settlement files minus payouts.
Quick check4 questions · +5 XP each

Reconciliation and the ledger

  1. Q1/4 A UPI payment has no callback after 30 seconds. Why not mark it failed and let the customer try again?

    Answer & explanation

    Answer: C

    A timeout is "unknown". Calling it failed invites a second payment for the same order. UNKNOWN, honest messaging and reconciliation avoid the double charge.

  2. Q2/4 A payment we marked FAILED appears in the next day's settlement file. The merchant's order is closed. What happens?

    Answer & explanation

    Answer: C

    The settlement file is the final word on whether money moved. With the order closed, the only right outcome is an automatic refund.

  3. Q3/4 Why keep a double-entry, append-only ledger instead of a balance column per merchant?

    Answer & explanation

    Answer: B

    In-place balances lose history and can lose concurrent updates. Postings that sum to zero, never edited, make every rupee traceable and every balance provable.

  4. Q4/4 Meera's ₹1,999 is captured with a ₹47.18 fee (2% plus 18% GST), then ₹500 is refunded. How much is settled to the merchant at T+2?

    Answer & explanation

    Answer: D

    ₹1,999 − ₹47.18 − ₹500 = ₹1,451.82. Settlement pays the merchant_payable balance, which already includes fees and refunds.

Deep Dive 5: Telling the Merchant, at Least Once (Week 2 Concepts: Webhook Delivery, Week 9: Noisy Neighbour)#

You

"Webhooks are where our exactly-once state meets the merchant's code. We can't make delivery exactly once over the internet; we make it at least once and give the merchant what they need to deduplicate."

Your turn · checkpoint 7not answered

The merchant's server is down

A large merchant's server starts returning 500s at 9 PM during a sale. Its webhooks pile up: thousands a minute. Meanwhile, ten thousand small merchants are receiving theirs normally. How do you keep retrying the big merchant without hurting anyone else, and what does the merchant's code need to do?

Think it through, or jot it down. Then compare.

One strong answer

  • A queue per merchant: each merchant's deliveries are independent, with their own concurrency limit. A failing merchant's retries back up in its own queue, not in a shared one.
  • Retry with exponential backoff for 24 hours: the documented behaviour is retries for 24 hours with exponential backoff, a 5-second response limit, and the webhook disabled after 24 hours of continuous failure, with the merchant alerted.
  • Every event has an id and a signature: the merchant verifies the signature (HMAC over the body with their secret) and deduplicates on the event id, because the same event can arrive more than once.
  • Order isn't guaranteed: the merchant must not assume payment.captured arrives after payment.authorized; it should treat events as hints and fetch the payment's current state when it matters.
  • Answer fast, work later: the merchant's handler should store the event and return 200 within 5 seconds; slow handlers time out and cause duplicates.

The Solution:

DiagramWebhook delivery, isolated per merchant
%% title: Webhook delivery, isolated per merchant
flowchart TD
  BUS@{ icon: "tech:kafka", label: "Kafka<br/>payment events" }
  FAN["Fan-out<br/>event → subscribed merchants"]:::service
  MQ@{ icon: "sd:queue", label: "Queue per merchant<br/>concurrency limit each" }
  SEND["Sender<br/>sign · POST · 5 s timeout"]:::hot
  RETRY@{ icon: "sd:timer", label: "Retry schedule<br/>exponential backoff, 24 h" }
  MS@{ icon: "sd:server", label: "Merchant server<br/>verify, dedupe on event id" }
  OFF["Disable after 24 h of failures<br/>alert the merchant"]:::bad
  BUS ==> FAN ==> MQ ==> SEND ==> MS
  SEND -.->|"non-2xx or timeout"| RETRY
  RETRY -.-> MQ
  RETRY -.->|"24 h"| OFF
  class BUS queue
  class MS client
  class RETRY cache
pythonDelivery · 64 lines
# webhooks/sender.py

"""
At-least-once webhook delivery. Each merchant has its own queue and
concurrency limit, so one failing merchant can't slow the others.
Every delivery is signed and carries a stable event id.
"""

import hashlib
import hmac
import json
import random
from dataclasses import dataclass

TIMEOUT_S = 5
RETRY_WINDOW_S = 24 * 3600
BASE_S, CAP_S = 30, 3600


@dataclass
class Delivery:
    event_id: str              # stable across retries: the merchant dedupes on it
    merchant_id: str
    url: str
    body: dict
    created_at: float
    attempt: int = 0


def signature(secret: bytes, raw: bytes) -> str:
    return hmac.new(secret, raw, hashlib.sha256).hexdigest()


def next_delay(attempt: int, rng: random.Random) -> float:
    """Exponential backoff with jitter, capped, so retries don't arrive in waves."""
    return min(CAP_S, BASE_S * 2 ** attempt) * (0.5 + rng.random() / 2)


class Sender:
    def __init__(self, http, secrets, queues, alerts, rng):
        self.http = http
        self.secrets = secrets
        self.queues = queues          # per-merchant queues with concurrency limits
        self.alerts = alerts
        self.rng = rng

    def deliver(self, d: Delivery, now: float) -> None:
        raw = json.dumps(d.body, separators=(",", ":")).encode()
        headers = {
            "X-Event-Id": d.event_id,
            "X-Signature": signature(self.secrets[d.merchant_id], raw),
        }
        try:
            r = self.http.post(d.url, data=raw, headers=headers, timeout=TIMEOUT_S)
            if 200 <= r.status < 300:
                return                                            # done; never resent
        except TimeoutError:
            pass                                                  # may have been processed: hence event ids
        if now - d.created_at >= RETRY_WINDOW_S:
            self.queues.disable(d.merchant_id)
            self.alerts.merchant(d.merchant_id, "webhook disabled after 24 hours of failures")
            return
        d.attempt += 1
        self.queues.schedule(d.merchant_id, d, run_at=now + next_delay(d.attempt, self.rng))
Quick check3 questions · +5 XP each

Webhooks

  1. Q1/3 Why give each merchant its own webhook queue and concurrency limit?

    Answer & explanation

    Answer: B

    Isolation per merchant turns one merchant's outage into that merchant's backlog. Everyone else keeps receiving events on time.

  2. Q2/3 A merchant's handler takes 8 seconds to process each webhook before answering. What will happen?

    Answer & explanation

    Answer: C

    The response limit is 5 seconds. The fix on the merchant's side is to store the event, answer 200 at once, and process it afterwards, deduplicating on the event id.

  3. Q3/3 The merchant receives payment.captured before payment.authorized. What should its code do?

    Answer & explanation

    Answer: D

    Ordering isn't guaranteed. The payment's state in the API is the truth; the webhook says "something changed, go look".

Phase 5 · Stage 5 of 7 · 5 min budget · 2 min read

Phase 5: Scaling and Edge Cases (5 minutes)#

Interviewer

"Walk me through a sale midnight, then tell me what breaks."

A Sale Midnight#

A big merchant's sale at midnight (illustrative)
  • Days before · Plan
    • The merchant shares expected volume; capacity for its slices is reserved with processors
    • Its traffic gets its own rate limit and connection pools, so its spike can't starve other merchants
  • 11:50 PM · Warm up
    • Router stats are fresh; probe traffic confirms every route is healthy
    • Deployments frozen; on-call watching the sale dashboard
  • 00:00 · The spike
    • Payments jump to many times normal for this merchant in seconds; the gateway's own work stays under 100 ms
    • Unknowns rise as some banks slow; the reconciler's schedule absorbs them
  • 00:15 · Card authentication slows at one issuer
    • The router moves that issuer's traffic to the next-best acquirer for that slice; probes keep testing
  • Next day · Reconcile and settle
    • Settlement files matched against the ledger; unknowns closed; late successes refunded or captured
    • At T+2, the merchant's settlement includes the night's captures minus fees and refunds

Edge Cases#

Edge cases
Edge case 1: Customer pays twice for one order

Problem: The first UPI payment was slow, so Meera paid again from another app; both succeeded.

Solution:

  • The order accepts one captured payment; the second success is captured by the state machine as a duplicate for that order and refunded automatically
  • While a payment is UNKNOWN, checkout says "we're confirming your payment, don't pay again" and offers no second attempt
Edge case 2: Callback arrives before our own record

Problem: A processor's success callback arrives a few milliseconds before the payment row's PENDING commit is visible.

Solution:

  • Callbacks for unknown or not-yet-pending payments are stored and retried for a short window, never dropped
  • The transition is guarded by state, so applying it late is safe
Edge case 3: A processor goes down completely

Problem: One acquirer stops answering at all.

Solution:

  • Its success rate falls to zero within seconds; the router marks every slice down and moves traffic to others
  • Payments already sent to it are UNKNOWN and reconciled from its status API or files once it's back
Edge case 4: Chargeback weeks later

Problem: A cardholder disputes a payment a month after settlement.

Solution:

  • A posting moves the amount from the merchant's payable (or a reserve) to "chargeback receivable"; the merchant is notified with evidence deadlines
  • If the merchant's balance is too low, future settlements are netted
Edge case 5: Stolen card testing

Problem: A script tries thousands of stolen card numbers at ₹1 on one small merchant.

Solution:

  • Velocity checks per card fingerprint, device, IP and merchant; challenge or block before the processor sees them
  • Under the RBI's authentication rules, issuers also apply their own risk-based checks on top of two-factor authentication
Quick check3 questions · +5 XP each

Scaling and edge cases

  1. Q1/3 Meera's first UPI payment was slow, she paid again, and both succeeded. What should happen?

    Answer & explanation

    Answer: B

    An order accepts one captured payment. The state machine recognises the second success for the same order and refunds it, without a support ticket.

  2. Q2/3 A processor's success callback arrives before the payment's PENDING state is committed. What does the gateway do with it?

    Answer & explanation

    Answer: C

    Dropping it would leave the payment unknown for minutes. Holding it briefly and applying the transition once the row is ready costs nothing and loses nothing.

  3. Q3/3 Why give a big merchant's sale its own rate limit and connection pools?

    Answer & explanation

    Answer: C

    One merchant's midnight is everyone else's normal Tuesday. Bulkheads per merchant keep the spike inside its own capacity.

Phase 6 · Stage 6 of 7 · 2 min

Phase 6: Monitoring and Operations#

You

"In payments, the most important metric isn't latency. It's whether every rupee is accounted for: success rate by slice, unknowns by age, and the daily three-way match."

Monitoring Dashboard#

Payments — sale night, 00:07 (illustrative)
Throughput and success
Metric Now
Payments a second (all merchants) 3,840
Success rate, cards / UPI / net banking 91.2% / 93.8% / 88.5%
Gateway's own latency p99 64 ms
Payments in UNKNOWN (oldest) 2,310 (4 min 10 s)
Routing
  • Slices marked down: 3 (issuer X via acquirer P1, two net-banking banks)
  • Probe traffic: 2% to down routes
Webhooks
  • Delivered within 5 s of the event: 99.4%
  • Merchants in backoff: 37; largest backlog: 18,200 events (one merchant returning 500s)
Money
  • Yesterday's three-way match (ledger, escrow statement, files): matched, 0 open items
  • Late successes today: 412, all captured or refunded
Alerts
  • 00:00:40 (amber) — Card success for issuer X via P1 at 61%, route down for 20 minutes
  • 00:03:12 (amber) — Merchant m_1180 webhook failures at 100%, retries backing off
  • 00:05:00 (green) — UNKNOWN count steady; reconciliation keeping up

SLOs for the Gateway#

SLOs for a payment gateway
SLO 1: Exactly one outcome
  • Target: 100% of payments reach a final state; none in UNKNOWN beyond the next settlement file
  • Error budget: 0
SLO 2: Money integrity
  • Target: ledger, escrow statement and processors' files match every day; open items resolved within 2 working days
SLO 3: Availability
  • Target: 99.99% of payment API requests succeed (excluding customer declines)
SLO 4: Merchant notification
  • Target: 99% of webhooks sent within 5 s of the state change; every event retried for 24 hours
SLO 5: Safe change
  • Target: every deployment canaried and judged before reaching the most critical merchants
You

"On that last SLO, Razorpay's engineering team has written about how they deploy: canaries that step through 1%, 2%, 5%, 10% and 15% of traffic with automated analysis at each step, and rollouts ordered by merchant criticality so the most critical merchants see a build last. That's the right instinct for a payment system: most incidents start with a change, not a crash."

Quick check3 questions · +5 XP each

Reading the dashboard

  1. Q1/3 2,310 payments are in UNKNOWN, the oldest 4 minutes 10 seconds. Is that an incident?

    Answer & explanation

    Answer: A

    UNKNOWN is a normal state at peak. The alarm is the age and the trend: payments stuck past the last status check, or a count that keeps growing.

  2. Q2/3 One merchant has 18,200 webhook events backed up. What is the effect on other merchants?

    Answer & explanation

    Answer: B

    Isolation per merchant keeps one merchant's outage inside its own backlog. Retries back off for 24 hours, then the webhook is disabled and the merchant alerted.

  3. Q3/3 Why is the daily three-way match an SLO with an error budget of zero open items left unresolved?

    Answer & explanation

    Answer: C

    The ledger, the escrow bank's statement and processors' files are three views of the same money. When they agree, nothing was lost; when they don't, a person investigates.

Final challenge · Stage 7 of 7 · 2 min

Final Challenge: Midnight Sale, One Slow Bank#

The interviewer leans back. "Let's put it all together."

Interviewer

"A large merchant's sale opens at midnight with twenty times its normal traffic. At 00:02, the card authentication pages of one big issuing bank slow to 20 seconds, and its UPI responses start timing out. Thousands of customers are mid-payment. Merchants' servers are getting hammered too. What do you do?"

The midnight brief
  • One merchant at 20× normal; everyone else normal
  • One issuing bank slow for cards (20 s authentication) and timing out on UPI
  • Customers retry when they see spinners
  • Some merchants' servers return errors under their own load
  • Nothing may be charged twice, and every payment must end in one final state
Your turn · checkpoint 8not answered

Contain the bank, protect the customer, keep the books

What changes in routing, in checkout, in reconciliation and in webhooks? What do customers of the slow bank see?

Think it through, or jot it down. Then compare.

One strong answer

  • Contain the merchant: its rate limit and pools are its own. Its 20× can use its reserved capacity and no more, so other merchants' payments stay at normal latency.
  • Routing can't fix an issuer: for cards from that bank, every acquirer reaches the same slow authentication page, so moving traffic doesn't help. The router learns that (all slices for the issuer are down) and stops flapping between acquirers.
  • Tell customers early: checkout shows a notice for that bank ("payments from this bank are slow right now") and suggests another method, before they start. That's cheaper and kinder than a 20-second spinner.
  • Unknowns grow, by design: UPI timeouts become UNKNOWN, with three status checks 90 seconds apart and the settlement file as the backstop. Customers see "confirming, don't pay again", and a second attempt for an order with an UNKNOWN payment is blocked.
  • Late money is handled: payments that turn out to have succeeded become late successes: captured if the order is open, otherwise refunded automatically.
  • Webhooks back off per merchant: merchants whose servers are failing get exponential retries in their own queues; nobody else waits.
  • The books still balance: every state change still posts its entries; tomorrow's three-way match proves it.
Quick check4 questions · +5 XP each

Final challenge · Quest check

  1. Q1/4 One issuing bank's card authentication is slow for every acquirer. Why doesn't routing help?

    Answer & explanation

    Answer: B

    Routing helps when one path is worse than another. When the issuer itself is slow, the fix is to tell its customers early and suggest another method.

  2. Q2/4 Customers of the slow bank keep tapping "Pay" again. What stops double charges?

    Answer & explanation

    Answer: C

    The order is the unit. While its payment is unknown, checkout says "confirming" and refuses a second attempt; if one slips through and both succeed, the duplicate is refunded.

  3. Q3/4 The sale's merchant is at 20× normal. What protects every other merchant?

    Answer & explanation

    Answer: C

    Bulkheads per merchant keep one sale's spike in its own compartment, for payments and for notifications.

  4. Q4/4 At 10,000 TPS with ~15 internal operations per payment, how many operations a second must the gateway handle?

    Answer & explanation

    Answer: C

    10,000 × 15 = 150,000. Sizing for payments alone would miss the routing, ledger, events and webhooks each payment creates.

Wrap-up · 5 min

Interview Conclusion#

Transcript
Interviewer

"Good. A few quick ones to finish."

Interviewer

"Why not use a distributed transaction across the gateway, the processor and the bank?"

You

"Because the bank isn't a participant we can lock. It answers when it answers, and sometimes only in tomorrow's file. So the design is a state machine with guarded transitions, an UNKNOWN state, and reconciliation against independent records. That's how you get exactly one outcome without anyone holding a lock across the internet."

Interviewer

"Where does 'exactly once' really live?"

You

"In two places we own: the idempotency record that commits with the effect, and the conditional state transition on one row. Everything that crosses a network, callbacks, status checks, webhooks, is at least once, and is made safe by those two."

Interviewer

"What's the single most important design decision?"

You

"Never turning silence into an answer. A timeout is UNKNOWN, not failed. That one rule is what stops a slow bank from becoming double charges, and the ledger plus the daily match is what proves we got every case right."

Summary: Concepts Applied from 10-Week Course#

Concepts from the 10-week course in payment-gateway design
  • Week 2: Failure-first design
    • Idempotency — keys stored with the effect, bound to a request hash
    • Timeouts — a timeout becomes UNKNOWN, never "failed"
    • Circuit breakers — routes marked down per slice, with probes to recover
    • Webhook delivery — at least once, signed, event ids, backoff for 24 hours
  • Week 3: Messaging and async processing
    • Transactional outbox — state change, ledger entries and event in one commit
    • Audit log — an append-only, double-entry ledger
  • Week 5: Consistency and coordination
    • Conditional state transitions on one row as the only writer
    • Saga orchestration — reconciliation drives every payment to a final state
  • Week 9: Multi-tenancy and security
    • Noisy neighbour — per-merchant limits, pools and webhook queues
    • A minimal card vault; tokens everywhere else
  • Week 10: Production readiness
    • SLOs for money integrity and exactly one outcome; canaried deployments

Why Razorpay Matters#

Why this is a different kind of hard problem
Exactly once across parties you don't control
  • The bank, the network and the merchant each have their own timing and their own records. "Exactly once" is built from idempotency, guarded transitions and reconciliation, not from a transaction
Silence is the normal failure
  • At a billion transactions a quarter, some payments always have no answer yet. Designing UNKNOWN as a state, and resolving it within NPCI's limits and through settlement files, is the core of the job
Money must be provable
  • The ledger, the escrow statement and processors' files must agree every day. A gateway that can't prove where every rupee is, isn't done

"The bank that never answered isn't the bug. The bug is the system that guessed."

Self-Assessment Checklist#

After studying this case study, you should be able to:

Architecture:

Architecture0/2 ready
  • Draw a gateway with a payment state machine, a vault, a router, adapters, a ledger and an outbox
  • Explain which steps are exactly once and which are at least once with deduplication

Correctness:

Correctness0/3 ready
  • Implement idempotent APIs whose key record commits with the effect
  • Model UNKNOWN and LATE_SUCCESS states and resolve them by reconciliation
  • Explain why status checks can't replace settlement-file reconciliation

Routing:

Routing0/1 ready
  • Route by success rate per (method, network, issuer, processor) with temporary downtime and probes

Money:

Money0/2 ready
  • Post balanced, append-only ledger entries for capture, fees, refunds and settlement
  • Compute a merchant's settlement from the ledger and prove it with a three-way match

Operations:

Operations0/2 ready
  • Deliver webhooks at least once with per-merchant isolation
  • Define SLOs for exactly one outcome and money integrity

Sources#

Company figures:

Razorpay engineering and documentation:

Regulation and the payments network:

Design note: the architecture, code, thresholds, pricing, rates and dashboards in this case study are an illustrative design for the problem, not a description of Razorpay's internal systems beyond what Razorpay has published in the sources above.

Further Reading#

Related systems to study:

  • Read

    India's UPI (Bonus Problem 1 in this series): the switch on the other side of every UPI payment, with its own deemed-success state

  • Read

    Visa / Mastercard (Bonus Problem 4): the card network path, authorisation and settlement

  • Read

    BookMyShow (Bonus Problem 16) and Swiggy (Bonus Problem 17): merchants that depend on unknown outcomes being resolved correctly

Foundational concepts (this course):

  • Read

    Week 2, Day 2 — Idempotency in Practice

  • Read

    Week 2, Day 3 — Circuit Breakers

  • Read

    Week 2, Day 4 — Webhook Delivery

  • Read

    Week 3, Day 2 — Transactional Outbox

  • Read

    Week 3, Day 5 — Audit Log System

  • Read

    Week 5, Day 3 — Saga Orchestration

  • Read

    Week 9, Day 2 — Noisy Neighbour

"A billion payments a quarter, and somewhere every minute a bank that hasn't answered. The gateway that earns trust isn't the one that never meets silence. It's the one that never mistakes silence for an answer."

End of Side quest 18 · Razorpay Payment Gateway
Clear the questFinish Razorpay.Stages clear as you read them. Marking complete unlocks after ~6 more min of reading.
  • 0/7stages cleared
design sheet · 6 diagrams · pdfRevise the whole design in ten minutesThe numbers, architecture, sequence, class and state diagrams, decisions and failure modes of this quest on a few printable pages.open the sheet →

New lessons by email

New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.

Discussion · …

Public: everyone reading this lesson sees comments. Be kind and keep it on the material.

Sign in to ask a question or answer one. Reading stays open to everyone.
loading the discussion…