A customer taps "Pay ₹1,000". Somewhere between your servers, a payment provider, a card network and a bank, the request times out. Did the money move? This is the question at the heart of every payment interview. The traffic is small next to a social network's; the bar for correctness is not. You must never charge twice, never lose a payment, and be able to prove where every rupee went.
The payment-system interview end to end: charge customers through payment providers using tokens, refund, pay sellers and reconcile, about a million payments a day with 120 a second at peak, where correctness beats availability. Each payment is a guarded state machine (marked PROCESSING before the PSP is called), success is committed with its ledger entries and an outbox row, and events reach the order service exactly when committed. Deep dives show why a timeout means unknown, one idempotency key per hop, a double-entry ledger that always sums to zero, and daily reconciliation against settlement files.
Phase 1: Requirements (5 minutes)#
"Do we process cards ourselves, or go through payment service providers (PSPs)?"
"Through PSPs. We use more than one."
"Good: then card numbers never touch our servers. The PSP's hosted form or SDK turns them into a token, which keeps most of our system out of PCI DSS scope. What we own is the state of each payment and the ledger."
- Pay-in: charge a customer for an order (cards, UPI, wallets) through a PSP
- Refunds: full or partial, any number of times up to the amount paid
- Ledger: record every movement of money, double-entry, immutable
- Pay-out: pay sellers their balance on a schedule, minus fees
- Reconciliation: match our records against each PSP's daily settlement report
- Status: the order service learns the outcome of every payment, exactly once
- Correctness over availability: better to fail a payment than charge twice or lose one
- Idempotency everywhere: every retry of any step is safe
- Auditability: every change traceable for years; nothing is edited or deleted
- Latency: a few seconds is acceptable (the PSP and bank dominate)
- Security: no raw card data stored; signed webhooks; least-privilege access to money movement
Phase 2: Estimation (5 minutes)#
- Payments: 1M/day ÷ 86,400 s ≈ 12/s on average, ~10x at a sale peak ≈ 120/s
- Each payment writes a payment record and about 4 ledger entries (more with fees, refunds, payouts)
- Ledger: 4M entries/day × ~200 B ≈ 800 MB/day → ~300 GB/year, kept for at least 7 years
- Calls to PSPs: ~120/s at peak, each taking 1–3 s, so a few hundred in flight at once
- Settlement reports: one file per PSP per day, up to ~1M lines each
"Nothing here needs a big cluster. One well-run relational database holds the payments and the ledger. Every hard problem in this design comes from the network between us and the PSP, not from volume."
Phase 3: API (3 minutes)#
POST /v1/payments
Idempotency-Key: order_8812-attempt-1
{ "order_id": "order_8812", "amount": 100000, "currency": "INR", "payment_method_token": "pm_tok_..." }
→ 201 { "payment_id": "pay_31", "status": "PROCESSING" } # amount in paise: ₹1,000.00
GET /v1/payments/pay_31 → 200 { "status": "SUCCEEDED", ... }
POST /v1/payments/pay_31/refunds Idempotency-Key: rf-1 { "amount": 25000 } → 201 { "refund_id": "rf_7" }
POST /v1/psp/{psp}/webhooks (from the PSP; signature verified) → 200Amounts are integers in the smallest unit (paise, cents). Floating point can't represent 0.1 exactly, and a ledger that's off by a fraction of a paisa never reconciles.
Phase 4: High-level design (10 minutes)#
%% title: Payment system: a state machine per payment, a double-entry ledger, reconciliation with providers
flowchart TD
CK@{ icon: "sd:cart", label: "Checkout / order service" }
PS["Payment service<br/>state machine · idempotency keys"]:::core
DB@{ icon: "tech:postgresql", label: "PostgreSQL<br/>payments · ledger · outbox" }
AD["PSP adapters<br/>one per provider"]:::service
PSP@{ icon: "sd:card", label: "Payment providers" }
BK@{ icon: "sd:bank", label: "Card networks · banks · UPI" }
K@{ icon: "tech:kafka", label: "Kafka · payment events" }
RC["Reconciliation<br/>daily, per provider"]:::service
PO["Payouts<br/>seller balances"]:::service
CK --> PS
PS --> DB
PS --> AD
AD -->|"charge(payment_id as key)"| PSP
PSP --> BK
PSP -.->|"signed webhook"| PS
DB -->|"outbox relay"| K
K --> CK
PSP -.->|"settlement file"| RC
RC --> DB
PO --> DB
class CK client
class DB store
class PSP,BK external
class K queue- Create. Checkout calls
POST /paymentswith an idempotency key. The payment service storespay_31in stateCREATED, unique by key: a repeat returns the same payment. - Send. It moves
pay_31toPROCESSINGbefore calling the PSP, then calls the PSP adapter, passingpay_31as the PSP's own idempotency key. - Result. The PSP answers (or later sends a signed webhook):
SUCCEEDEDorFAILED. The service applies the transition only if the payment is stillPROCESSING. A duplicate webhook changes nothing. - Record. In the same database transaction as the state change, it writes the ledger entries and an outbox row.
- Tell the world. The outbox relay publishes
payment.succeededto Kafka; the order service marks the order paid. - Reconcile. Next day the PSP's settlement file is matched line by line against the ledger.
Phase 5: Deep dives (15 minutes)#
Deep dive 1: Never charging twice#
The timeout
The payment service calls the PSP to charge ₹1,000 and gets no answer within 10 seconds. The customer is waiting. Should it retry the charge?
One strong answer
- Not blindly. A timeout means unknown, not failed: the PSP may have charged the card and the reply was lost. Retrying as a new charge could bill the customer twice.
- Retry with the same key. Because
pay_31was sent as the PSP's idempotency key, a retry with that key returns the original result if the first call went through, and performs the charge only if it didn't. - Or ask. Mark the payment as unknown internally, show "processing" to the customer, and query the PSP for
pay_31's status, or wait for its webhook. A background job resolves stuck payments; reconciliation is the last word.
The full defence is layered, one idempotency key per hop:
Client → us
The checkout's Idempotency-Key is unique in our payments table. A double click or a retried request returns the existing payment.
Us → PSP
Our payment_id is the PSP's idempotency key. A retried charge is deduplicated by the PSP.
PSP → us
Webhooks carry the PSP's event ID; we store processed IDs and verify the signature. Duplicates and forgeries change nothing.
Us → order service
Events leave through the transactional outbox, so "payment succeeded" is published if and only if it was committed, and consumers dedupe on the event ID.
The state machine is the other half. Transitions are written as conditional updates (UPDATE payments SET status='SUCCEEDED' WHERE id='pay_31' AND status='PROCESSING'), so whichever arrives first, the API reply or the webhook, wins, and the second is a no-op. A payment never goes from SUCCEEDED back to anything; refunds are new records with their own state machine.
Deep dive 2: The ledger and reconciliation#
A balance column (seller.balance += 970) is easy to write and impossible to audit: when it's wrong, you can't say why. Payment systems use double-entry bookkeeping instead: every movement of money is a set of entries that sum to zero, written once and never changed.
| Event | Account | Debit (₹) | Credit (₹) |
|---|---|---|---|
| Customer pays ₹1,000 for order_8812 | PSP receivable | 1,000 | |
| Seller payable (seller_77) | 970 | ||
| Platform fee revenue | 30 | ||
| PSP settles next day, keeping a ₹20 fee | Bank account | 980 | |
| PSP fees expense | 20 | ||
| PSP receivable | 1,000 |
Every transaction's debits equal its credits, so the ledger as a whole always sums to zero, and any balance (what we owe seller_77, what the PSP owes us) is a sum over entries. A bug shows up as a ledger that doesn't balance, not as silently wrong money. Corrections are new entries that reverse old ones, never edits.
Reconciliation closes the loop with the outside world:
Match
Load the PSP's settlement file and match each line to our payment by payment_id (and amount and currency).
Classify mismatches
- Missing on our side: the PSP charged but we have no success. Usually a lost webhook: apply it.
- Missing on their side: we think it succeeded but the PSP has no settlement. Investigate; possibly reverse.
- Amount or fee mismatch: a partial capture, a currency conversion, or an unexpected fee.
Resolve
Known patterns are fixed automatically with correcting ledger entries; the rest go to a finance operations queue. A day closes only when every line is matched or explained.
Phase 6: Trade-offs and failure modes (5 minutes)#
PSP timeout
Unknown, not failed: retry with the same key or query status; never create a new charge for the same payment.
Webhook lost
A sweeper queries the PSP for payments stuck in PROCESSING after a few minutes; reconciliation catches anything left the next day.
PSP outage
Route new payments to a second PSP (each payment stays with the PSP it started on). Payments in flight on the failed PSP resolve when it recovers.
Database failover
Synchronous replication to a standby, so no committed payment or ledger entry is lost. Payments pause during failover rather than risk inconsistency.
A bug double-credits sellers
The ledger doesn't balance against the PSP receivable; reconciliation flags it before payouts run. Reversing entries fix it, with a full audit trail.
| Decision | We chose | We gave up |
|---|---|---|
| Card data | PSP tokens, no raw card numbers | Full control over card processing |
| Retries | Idempotency key on every hop | Simple fire-and-retry calls |
| Money records | Append-only double-entry ledger | A simple, editable balance column |
| Truth | Daily reconciliation with each PSP | Trusting our own records alone |
| Priority | Fail rather than risk a double charge | Some availability during incidents |
Go deeper in the course#
- Week 2, Day 2: Idempotency in practice: the idempotency key pattern, the network-timeout problem and the double-click problem, all central to payments
- Week 2, Day 1: Timeout hell: why a timeout means "unknown", and how to set them
- Week 2, Day 4: Webhook delivery system: signatures, retries and deduplication of webhooks
- Week 3, Day 2: Transactional outbox: publishing "payment succeeded" exactly when it's committed
- Week 5, Day 2: The saga pattern: order, payment and inventory as a saga with compensations
- Week 3, Day 5: Audit log system: append-only, tamper-evident records
- Week 9, Day 5: Security architecture: protecting sensitive data, PCI scope and access control
- Side quest: India's UPI: an interoperable real-time payment network
- Side quest: Visa/Mastercard: what happens beyond the PSP, through card networks and banks
Check yourself#
Payment system
Q1/4 The call to the PSP times out after 10 seconds. What does that tell you about the charge?
Answer & explanation
Answer: C
A lost reply looks exactly like a lost request. Retry with the same idempotency key or query the status; never start a new charge.
Q2/4 Both the PSP's API reply and its webhook report success for pay_31. How do you avoid applying it twice?
Answer & explanation
Answer: B
Transitions are guarded by the current state, so whichever arrives first wins and the duplicate changes nothing.
Q3/4 Why use a double-entry ledger instead of a balance column per seller?
Answer & explanation
Answer: D
Balances are derived from immutable entries; a bug produces a ledger that doesn't balance, and corrections leave an audit trail.
Q4/4 Why are amounts stored as integers in paise rather than as floating-point rupees?
Answer & explanation
Answer: A
0.1 has no exact binary float representation. Summing millions of entries in floats drifts by fractions of a paisa, and the ledger never reconciles.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.