Himanshu Kukreja
0%

Bonus Problem 11: IRCTC Tatkal Ticket Booking System

The Digital Lottery That Pits 2 Million Passengers Against 500 Seats β€” Every Single Day


πŸš† How Do You Sell 500 Seats to 2 Million People in 60 Seconds β€” Without Selling One Seat Twice?

It is 9:59:58 AM. Across India, in trains, buses, cyber cafΓ©s, offices, and bedrooms, two million fingers hover over the "Search" button. They are all trying to do the same thing: book a Tatkal ticket on the same popular train, for the same date, in the same class.

At 10:00:00 AM, the gates open. Ten seconds later, every seat is gone. Ninety-nine point nine seven five percent of the people who tried will fail β€” and they need to fail cleanly, instantly, and fairly. Not one seat can be sold twice. Not one rupee can be double-charged. And the 0.025% who succeed need their booking, and only their booking, confirmed.

This happens every day, on hundreds of trains, across India. This is the system we'll design today.

THE NUMBERS THAT DEFINE INDIAN RAILWAYS' E-TICKETING (2025)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                          β”‚
β”‚   AVERAGE TICKETS/DAY            RECORD SINGLE DAY                       β”‚
β”‚   ────────────────────           ──────────────────                      β”‚
β”‚   14.5+ Lakh (1.45M)             18.4 Lakh (1.84M) β€” 19 Aug 2025         β”‚
β”‚                                                                          β”‚
β”‚   RECORD TICKETS/MINUTE          PEAK CONCURRENT USERS                   β”‚
β”‚   ───────────────────            ─────────────────────                   β”‚
β”‚   37,410 β€” 16 Aug 2025           3+ Lakh (300,000+) sustained            β”‚
β”‚   (10 AM Tatkal window)          800K-1M attempt login at 10 AM          β”‚
β”‚                                                                          β”‚
β”‚   ENQUIRIES vs BOOKINGS          TATKAL WINDOWS                          β”‚
β”‚   ───────────────────            ───────────────                         β”‚
β”‚   ~4 Lakh enquiries/min          AC classes:     10:00 AM                β”‚
β”‚   vs ~32,000 bookings/min        Non-AC classes: 11:00 AM                β”‚
β”‚   (12:1 read-to-write ratio)                                             β”‚
β”‚                                                                          β”‚
β”‚   SUSPICIOUS IDs BLOCKED (2025)  SUSPICIOUS PNRs FLAGGED                 β”‚
β”‚   ──────────────────────────     ───────────────────────                 β”‚
β”‚   3.03 Crore (30.3M) deactivated  4.18 Lakh (418,000)                    β”‚
β”‚   6.05 Crore (60.5M) revalidated                                         β”‚
β”‚                                                                          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

For context:
β€’ A single popular train's Tatkal quota can be as small as 400-600 seats
β€’ On a hot route, that quota can attract over a million hopeful clicks
β€’ The system must reject 99%+ of requests in milliseconds, not seconds
β€’ One wrong seat allocation means a real family stranded at a real station

This is the system we'll design today.


The Interview Begins

You're interviewing at a company building ticketing infrastructure for a national transit agency. The principal architect leans forward:

Interviewer: "Everyone assumes booking a train ticket is 'just a database write.' It's not. I want you to design India's Tatkal ticket booking system β€” the part of Indian Railways' IRCTC platform that opens exactly 500 seats to millions of people at a precise second, every single morning."

╔══════════════════════════════════════════════════════════════════════════╗
β•‘                                                                          β•‘
β•‘           Design a High-Contention Ticket Booking System                 β•‘
β•‘                                                                          β•‘
β•‘   At exactly 10:00:00 AM, open booking for a Tatkal quota of ~500        β•‘
β•‘   seats on a single popular train. Roughly 2 million people will         β•‘
β•‘   attempt to book within the first minute.                               β•‘
β•‘                                                                          β•‘
β•‘   Requirements:                                                          β•‘
β•‘   β€’ Zero double-booking β€” a seat is sold to exactly one passenger        β•‘
β•‘   β€’ Zero double-charging β€” a payment is captured exactly once            β•‘
β•‘   β€’ Fair, fast rejection for the ~99.975% who don't get a seat           β•‘
β•‘   β€’ End-to-end booking (search β†’ seat β†’ pay β†’ PNR) in seconds            β•‘
β•‘   β€’ Resilient to script-based bots and bulk-booking agents               β•‘
β•‘   β€’ The other 12,999 trains running that day must be unaffected          β•‘
β•‘                                                                          β•‘
β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•

Interviewer: "Most candidates think this is a database problem. It's actually an admission-control problem wearing a database's clothes. Show me you understand the difference."


Phase 1: Requirements Clarification (5 minutes)

You: "Before I design anything, I want to understand exactly where the contention lives."

Your Questions

You: "First β€” is the entire IRCTC platform this contended, or is it specific trains?"

Interviewer: "Great distinction. Most of IRCTC's ~14.5 lakh tickets a day are boring β€” general quota, booked hours or days in advance, spread evenly across ~13,000 trains. The chaos is concentrated: a handful of popular routes (say, a Rajdhani or a Vande Bharat between two big cities) have a Tatkal quota β€” a small block of seats released exactly 24 hours before departure, at a fixed time. That's where 2 million people converge on 500 seats."

You: "So the seat inventory itself β€” how is it structured? Is it one row per seat, or something else?"

Interviewer: "Think of inventory as a pool: one bucket per (train, date, class, quota). A given train running tomorrow, in 3AC, under the Tatkal quota, might have exactly 486 seats in its pool. Every booking is really an allocation from a pool, followed by assigning a specific berth within it."

You: "What happens if a payment succeeds but the app crashes before showing the PNR? Or if a user's card is charged but the seat was already taken by someone faster?"

Interviewer: "Both must be handled with zero ambiguity. If we can't confirm a seat, the money must come back automatically β€” no support ticket required. And a user should never see 'payment successful' for a seat that isn't theirs."

You: "How do you stop travel agents and bots from buying up the Tatkal quota in the first few seconds and reselling tickets?"

Interviewer: "This is a real, ongoing arms race. As of mid-2025, Indian Railways mandated Aadhaar-based OTP authentication for Tatkal bookings and banned authorized agents from booking in the first 30 minutes of the window. Your design should assume adversarial traffic, not just heavy traffic."

You: "Last one β€” is search/availability-checking the same code path as booking?"

Interviewer: "Absolutely not, and that's a trap many candidates fall into. Roughly 4 lakh people check availability every minute during peak; only ~32,000 tickets get booked in that same minute. If your search path and your booking path share infrastructure, the search traffic alone will take down bookings."

Functional Requirements

1. SEARCH & AVAILABILITY
   β€’ Search trains by source, destination, date, class
   β€’ Show live-ish seat/berth availability per quota
   β€’ Show fare, route, duration

2. BOOKING
   β€’ Reserve seats from a (train, date, class, quota) pool
   β€’ Collect passenger details (name, age, gender, berth preference)
   β€’ Aadhaar-OTP-based identity verification for Tatkal
   β€’ Capture payment (UPI, card, net banking, wallet)
   β€’ Generate a PNR (Passenger Name Record) on confirmation

3. INVENTORY STATES
   β€’ CNF (Confirmed) β€” allocated a real berth
   β€’ RAC (Reservation Against Cancellation) β€” shares a berth, guaranteed seat
   β€’ WL (Waitlisted) β€” ordered position, no seat yet

4. POST-BOOKING
   β€’ Cancellation with tiered refund rules
   β€’ Automatic promotion: WL β†’ RAC β†’ CNF as seats free up
   β€’ Chart preparation ~4 hours before departure (final passenger list)
   β€’ SMS / push notification for status changes

5. ANTI-ABUSE
   β€’ One Tatkal booking per authenticated user per opening window
   β€’ Agent booking restriction in the first 30 minutes
   β€’ Bot / scripted-traffic detection

Non-Functional Requirements

SCALE (for one hot train's Tatkal window)
β€’ ~2,000,000 booking attempts within 60 seconds
β€’ ~500 seats available to allocate
β€’ Contention ratio: 4,000 requests for every 1 seat

SCALE (platform-wide, real IRCTC figures)
β€’ 14.5+ lakh tickets/day average, 18.4 lakh on record days
β€’ 37,410 tickets booked in a single minute (record, Aug 2025)
β€’ ~4 lakh availability checks/minute vs ~32,000 bookings/minute
β€’ 300,000+ concurrent users sustained during Tatkal windows

LATENCY
β€’ Admission decision (in queue / rejected): < 200ms
β€’ Seat hold confirmation: < 1 second
β€’ End-to-end booking (search already done β†’ PNR): < 15 seconds

CONSISTENCY
β€’ Zero oversell: never allocate the same berth twice
β€’ Exactly-once payment capture per booking attempt
β€’ Every debited rupee is either tied to a confirmed PNR or refunded

AVAILABILITY
β€’ A single hot train must never degrade booking for other trains
β€’ Search/enquiry traffic must never degrade the booking path

SECURITY
β€’ Identity-verified booking (Aadhaar OTP) for Tatkal
β€’ Detect and throttle scripted / bot traffic in real time
β€’ Rate-limit and flag bulk-booking patterns

Phase 2: Back of the Envelope Estimation (5 minutes)

You: "Let me size this precisely β€” because the numbers reveal why this is an admission-control problem, not a storage problem."

Traffic Estimation

THE FUNNEL FOR ONE HOT TRAIN

Booking attempts (60-second window):        2,000,000
Available seats:                                  500
Contention ratio:                             4,000 : 1

If arrivals were perfectly smooth over 60s:
  Average request rate:                     ~33,300 req/sec

Reality β€” Tatkal traffic is NOT smooth. Most of it lands
in the first 2-5 seconds, because users are refreshing a
page and hitting the same clock:

  Requests in first 5 seconds (~70% of total):   1,400,000
  Effective peak rate:                         ~280,000 req/sec

  For comparison, IRCTC's ENTIRE PLATFORM peak
  (all trains, nationally) is ~32,000 bookings/min
  = ~530 bookings/sec. One hot train's opening burst
  can exceed the platform's national write-throughput
  by 500x if left unthrottled.

PER BOOKING ATTEMPT, MULTIPLE BACKEND OPERATIONS:
β”œβ”€β”€ Session / login validation:      1 lookup
β”œβ”€β”€ Availability read:                1-2 reads
β”œβ”€β”€ Seat hold attempt:                1 write (mostly rejected)
β”œβ”€β”€ Passenger form + Aadhaar OTP:     1-2 calls (only for winners)
β”œβ”€β”€ Payment initiate + webhook:       2 calls (only for winners)
β”œβ”€β”€ PNR write + SMS:                  1-2 writes (only for winners)

Effective backend ops/sec at peak:   ~300,000+ for this ONE train

Storage Estimation

THIS IS THE COUNTERINTUITIVE PART: STORAGE IS TRIVIAL.

Per PNR record:
β”œβ”€β”€ PNR number:                8 bytes
β”œβ”€β”€ Train, date, class, quota: ~40 bytes
β”œβ”€β”€ Passenger details (Γ—1-6):  ~300 bytes
β”œβ”€β”€ Berth allocations:         ~60 bytes
β”œβ”€β”€ Payment reference:         ~40 bytes
β”œβ”€β”€ Timestamps, status:        ~40 bytes
└── Total:                     ~500-700 bytes

500 successful bookings for this train:      ~350 KB
1,999,500 failed/rejected attempts:           0 bytes stored
  (rejections should never touch durable storage)

Platform-wide, daily:
β”œβ”€β”€ ~1.45M confirmed PNRs Γ— 700B:            ~1 GB/day
β”œβ”€β”€ Waitlist + cancellation history:         ~2-3 GB/day
β”œβ”€β”€ Enquiry/search logs (analytics tier):    tens of GB/day
└── 7-year audit retention (bookings only):  ~2.5 TB

CONCLUSION: The bottleneck is never disk. It is the
number of requests that must be evaluated, rejected,
or serialized against 500 rows in the same second.

Key Metrics Summary

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Metric                          β”‚  Value                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Requests for this train (60s)   β”‚  2,000,000                   β”‚
β”‚  Seats available                 β”‚  500                         β”‚
β”‚  Success rate                    β”‚  0.025%                      β”‚
β”‚  Peak instantaneous rate         β”‚  ~280,000 req/sec            β”‚
β”‚  Storage for winners             β”‚  ~350 KB                     β”‚
β”‚  Storage for losers              β”‚  0 bytes (rejected in flight)β”‚
β”‚  National record: tickets/min    β”‚  37,410                      β”‚
β”‚  National: enquiry-to-booking    β”‚  ~12 : 1                     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Phase 3: High-Level Design (10 minutes)

You: "The core insight: this system needs two completely separate planes β€” a cheap, cacheable, horizontally-scalable read plane for search/enquiry, and a tiny, tightly-serialized write plane for the actual seat allocation. Mixing them is how real ticketing platforms fall over."

System Architecture

                          IRCTC PLATFORM ARCHITECTURE

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                          LAYER 1: CLIENT SURFACES                         β”‚
β”‚                                                                           β”‚
β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”       β”‚
β”‚   β”‚  Website   β”‚   β”‚ Rail Connectβ”‚  β”‚ PRS Counterβ”‚   β”‚  Agent     β”‚       β”‚
β”‚   β”‚            β”‚   β”‚   (App)     β”‚  β”‚ (offline)  β”‚   β”‚  Terminals β”‚       β”‚
β”‚   β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜       β”‚
β”‚         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜              β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     LAYER 2: EDGE / ADMISSION CONTROL                     β”‚
β”‚                                                                           β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”‚
β”‚  β”‚  READ PLANE (cheap)       β”‚      β”‚  VIRTUAL WAITING ROOM          β”‚    β”‚
β”‚  β”‚  ──────────────────       β”‚      β”‚  ─────────────────────         β”‚    β”‚
β”‚  β”‚  CDN + cache for search,  β”‚      β”‚  Issues queue tokens,          β”‚    β”‚
β”‚  β”‚  fares, availability      β”‚      β”‚  admits users at a rate the    β”‚    β”‚
β”‚  β”‚  snapshots (refreshed     β”‚      β”‚  write plane can actually      β”‚    β”‚
β”‚  β”‚  every 1-2 sec)           β”‚      β”‚  absorb. Rejects/queues the    β”‚    β”‚
β”‚  β”‚  Absorbs ~4 lakh/min      β”‚      β”‚  ~99.975% who can't win.       β”‚    β”‚
β”‚  β”‚  enquiry traffic here     β”‚      β”‚                                β”‚    β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                       β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    LAYER 3: BOOKING CORE (WRITE PLANE)                    β”‚
β”‚                                                                           β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚
β”‚  β”‚  Session &   β”‚  β”‚  Seat Inventory    β”‚  β”‚  Booking Saga            β”‚   β”‚
β”‚  β”‚  Identity    β”‚  β”‚  Service           β”‚  β”‚  Orchestrator            β”‚   β”‚
β”‚  β”‚  (Aadhaar    β”‚  β”‚  ────────────────  β”‚  β”‚  ───────────────         β”‚   β”‚
β”‚  β”‚  OTP, device β”‚  β”‚  One pool per      β”‚  β”‚  Hold β†’ passenger        β”‚   β”‚
β”‚  β”‚  binding)    β”‚  β”‚  (train,date,      β”‚  β”‚  details β†’ payment β†’     β”‚   β”‚
β”‚  β”‚              β”‚  β”‚  class,quota).     β”‚  β”‚  PNR, or compensate      β”‚   β”‚
β”‚  β”‚              β”‚  β”‚  Single-writer per β”‚  β”‚  (release + refund)      β”‚   β”‚
β”‚  β”‚              β”‚  β”‚  pool = no oversellβ”‚  β”‚                          β”‚   β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”‚
β”‚         β”‚                    β”‚                          β”‚                 β”‚
β”‚         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β”‚
β”‚                                          β–Ό                                β”‚
β”‚                            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                    β”‚
β”‚                            β”‚  Fraud / Risk Engine    β”‚                    β”‚
β”‚                            β”‚  Bot detection, velocityβ”‚                    β”‚
β”‚                            β”‚  checks, agent-window   β”‚                    β”‚
β”‚                            β”‚  restriction            β”‚                    β”‚
β”‚                            β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     LAYER 4: SETTLEMENT & DOWNSTREAM                      β”‚
β”‚                                                                           β”‚
β”‚   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚   β”‚  Payment      β”‚  β”‚  PNR /       β”‚  β”‚  Waitlist/RAC  β”‚  β”‚  SMS /     β”‚ β”‚
β”‚   β”‚  Gateways     β”‚  β”‚  Ticketing DBβ”‚  β”‚  Promotion &   β”‚  β”‚  Push      β”‚ β”‚
β”‚   β”‚  (UPI, cards, β”‚  β”‚  (active-    β”‚  β”‚  Chart Prep    β”‚  β”‚  Notify    β”‚ β”‚
β”‚   β”‚  netbanking)  β”‚  β”‚  passive)    β”‚  β”‚                β”‚  β”‚            β”‚ β”‚
β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

KEY INSIGHT:
β€’ The Virtual Waiting Room's ONLY job is to protect Layer 3 from Layer 1's volume
β€’ The Seat Inventory Service for a hot pool is intentionally NOT horizontally
  scaled β€” it is a single, fast, serialized writer, because correctness for
  500 rows matters infinitely more than throughput for 500 rows

Data Flow: Booking a Tatkal Ticket

You: "Let's trace Priya, who wants seat #501 of 500 possible winners..."

TATKAL BOOKING FLOW β€” PRIYA vs 1,999,999 OTHERS

  LANES:  [A] Priya's App   [W] Waiting Room   [B] Booking Core   [P] Payment/PNR

  09:59:50  [A] Pre-fetches search results, keeps session open

  ------------------------------------------------------------------------
  STEP 1   10:00:00   [A] --> [W]   "Request a queue token"

  STEP 2              [W] --> [A]   "Token issued β€” position 84,213,
                                      estimated wait 2 seconds"

  STEP 3              [W] --> [B]   "Admitted β€” rate set by whatever
                                      Booking Core can absorb right now"

  STEP 4              [A] --> [B]   "Select seat + hold"
                                      --> reply: "3AC, pool has 12 seats left"

  STEP 5              [B] --> [A]   "Hold granted (TTL 5 minutes)"

  STEP 6              [A] --> [B]   "Passenger details + Aadhaar OTP"

  STEP 7              [A] --> [P]   "Initiate payment"

  STEP 8              [P] --> [A]   "Payment success"

  STEP 9              [B] --> [A]   "PNR confirmed: 2847193651 β€”
                                      Seat 3A, Berth 24"
  ------------------------------------------------------------------------

TOTAL TIME: ~8-12 seconds for a winner

MEANWHILE, IN PARALLEL:
β€’ 1,999,500 other requests are rejected at STEP 2 or STEP 3 in < 200ms
β€’ They never reach the Seat Inventory Service at all
β€’ This is the whole point: fail cheap, fail fast, fail at the edge

Phase 4: Deep Dives (20 minutes)

Deep Dive 1: 500 Seats, Millions of Hands β€” The Hottest Rows in the Database (Week 1 Concepts: Hot Keys & Skew)

You: "The single biggest risk in this design isn't the database being too small β€” it's 500 rows becoming, for 60 seconds, the hottest data in the entire country's transactional systems."

The Challenge:

THE HOT KEY PROBLEM

A normal database row gets touched a few times a second.
This train's Tatkal pool gets touched 280,000 times a second.

If we naively implement this as:

  SELECT available_seats FROM pool WHERE train_id=X AND quota='TQ'
  UPDATE pool SET available_seats = available_seats - 1
      WHERE train_id=X AND quota='TQ' AND available_seats > 0

...every single one of those 280,000 requests/sec fights for the
SAME row lock. The database doesn't process bookings anymore β€”
it processes lock queue churn. Latency explodes, timeouts cascade,
and even the 500 legitimate winners can't get through the noise.

Splitting the row into shards (a classic hot-key fix) doesn't
fully help here either: the total pool is only 500 seats. Split
into 10 shards of 50 seats each, and you just create 10 hot rows
instead of 1 β€” plus now you need cross-shard rebalancing when one
shard empties before another.

The Solution:

SEPARATE "CAN I EVEN TRY?" FROM "GIVE ME THE SEAT"

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚   STAGE 1: FAST ADMISSION CHECK (Redis, in-memory, single-digit ms)     β”‚
β”‚   ────────────────────────────────────────────────────────────────      β”‚
β”‚                                                                         β”‚
β”‚   A single atomic counter per pool: tatkal:12301:2026-08-10:3A          β”‚
β”‚   Value starts at 500. Every request runs one atomic Lua script:        β”‚
β”‚                                                                         β”‚
β”‚      if counter > 0: counter -= 1; return TICKET_TO_TRY                 β”‚
β”‚      else:            return SORRY_TRY_ANOTHER_TRAIN                    β”‚
β”‚                                                                         β”‚
β”‚   This single operation is O(1), lock-free from the caller's            β”‚
β”‚   perspective, and Redis can execute ~100,000+ of these per second      β”‚
β”‚   on a single core. It rejects 1,999,500 requests almost for free.      β”‚
β”‚                                                                         β”‚
β”‚   STAGE 2: DURABLE ALLOCATION (only for the ~500 who passed Stage 1)    β”‚
β”‚   ────────────────────────────────────────────────────────────────      β”‚
β”‚                                                                         β”‚
β”‚   Only requests holding a "ticket to try" reach the real database.      β”‚
β”‚   At this reduced volume (500, not 2,000,000), a single-writer,         β”‚
β”‚   fully serialized allocator can safely assign real berth numbers       β”‚
β”‚   with a normal transactional UPDATE β€” no contention left to fight.     β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

WHY THIS WORKS:
β€’ The hot key (the counter) is intentionally simple: one integer,
  one atomic decrement, no joins, no side effects
β€’ 99.975% of load is absorbed by something that isn't a database at all
β€’ The database only ever sees traffic proportional to actual inventory
# inventory/hot_pool_admission.py

"""
Fast admission control for a single hot (train, date, class, quota) pool.

Design principle: the pool counter must answer ONE question, as fast as
possible, for as many callers as possible. It does not allocate berths,
does not touch passenger data, and does not talk to a durable database.
"""

from dataclasses import dataclass
from typing import Optional
import time


ADMISSION_SCRIPT = """
-- KEYS[1]: pool counter key, e.g. "pool:12301:2026-08-10:3A:TQ"
-- ARGV[1]: this request's unique attempt id (for idempotent retries)

local remaining = tonumber(redis.call('GET', KEYS[1]))
if remaining == nil then
    return -2  -- pool not initialized / booking not open yet
end

if remaining <= 0 then
    return -1  -- sold out, reject immediately
end

redis.call('DECR', KEYS[1])
redis.call('SADD', KEYS[1] .. ':admitted', ARGV[1])
return remaining - 1
"""


@dataclass
class AdmissionResult:
    admitted: bool
    reason: str
    remaining_hint: int = 0


class HotPoolAdmissionGate:
    """
    Guards a single (train, date, class, quota) pool from the full
    force of Tatkal traffic. This is the ONLY thing 2 million
    requests are allowed to touch directly.
    """

    def __init__(self, redis_client, metrics):
        self.redis = redis_client
        self.metrics = metrics
        self._script_sha = None

    async def initialize_pool(self, pool_key: str, seat_count: int):
        """Called once, moments before the Tatkal window opens."""
        await self.redis.set(pool_key, seat_count, nx=True)

    async def try_admit(self, pool_key: str, attempt_id: str) -> AdmissionResult:
        """
        The ONLY call 2 million requests make. Must resolve in
        single-digit milliseconds under load.
        """
        result = await self.redis.evalsha(
            self._script_sha, keys=[pool_key], args=[attempt_id]
        )

        if result == -2:
            self.metrics.increment("admission_not_open", pool_key)
            return AdmissionResult(False, "BOOKING_NOT_OPEN")

        if result == -1:
            self.metrics.increment("admission_sold_out", pool_key)
            return AdmissionResult(False, "SOLD_OUT")

        self.metrics.increment("admission_granted", pool_key)
        return AdmissionResult(True, "PROCEED_TO_ALLOCATION", remaining_hint=result)

Deep Dive 2: The Virtual Waiting Room β€” Taming the Thundering Herd (Week 3 & 4 Concepts: Backpressure & Flow Control, Thundering Herd)

You: "A hot pool admission gate solves contention inside the system. But 2 million simultaneous connections can crush the edge before a single request even reaches that gate. We need a front door that only opens as fast as the house behind it can hold people."

The Challenge:

THE THUNDERING HERD, LITERALLY

At 09:59:59, hundreds of thousands of browser tabs and app
instances are all sitting on a "Search" or "Book Now" button,
synchronized to the same clock. At 10:00:00.000, they all fire
at once.

This is not "high average load." It is a near-instantaneous
step function: from near-zero to 200,000+ req/sec in under a
second. No amount of auto-scaling reacts fast enough β€” by the
time new servers are up, the herd has already stampeded through
the old ones and knocked them over.

Naive load balancing just spreads the stampede across more
servers, all of which still try to hit the same 500-seat pool
at the same instant.

The Solution:

A VIRTUAL WAITING ROOM WITH ADAPTIVE ADMISSION

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                        β”‚
β”‚   Client              Waiting Room               Booking Core          β”‚
β”‚                                                                        β”‚
β”‚   Connects  ─────▢  Assigned a fair,       ◀── Reports live health:    β”‚
β”‚   before             randomized queue           CPU, DB latency,       β”‚
β”‚   10:00:00           position (NOT simply       payment gateway        β”‚
β”‚                      first-request-wins,         success rate          β”‚
β”‚                      which would reward               β”‚                β”‚
β”‚                      the fastest scripts,              β–Ό               β”‚
β”‚                      not the fastest humans)     Admission rate        β”‚
β”‚                                                   adapts every 1s:     β”‚
β”‚   Polls or                                        "System healthy β†’    β”‚
β”‚   holds a         ◀── Admitted in small           admit 600/sec"       β”‚
β”‚   WebSocket           controlled batches,         "DB latency rising β†’ β”‚
β”‚                       proportional to what         throttle to 200/sec"β”‚
β”‚                       downstream can absorb                            β”‚
β”‚                                                                        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

KEY DESIGN DECISIONS:

1. QUEUE POSITION IS RANDOMIZED WITHIN SHORT WINDOWS, NOT PURE FIFO
   Pure "first byte wins" rewards whoever has the fastest script,
   not the fastest human β€” exactly the bot behavior we want to
   discourage. Bucket arrivals into small time windows (e.g. 250ms)
   and shuffle within each bucket.

2. ADMISSION RATE IS ADAPTIVE, NOT FIXED
   A fixed "admit 1,000/sec" is either too conservative (wasting
   capacity) or too aggressive (overloading the booking core the
   moment DB latency ticks up). Feed back real signals every second.

3. ONE ACTIVE QUEUE SLOT PER SESSION
   A user's session is bound to their device (Week 1 concepts:
   session store design). Opening 10 tabs or automating 10 requests
   from one account must not yield 10 queue positions.

4. REJECTED USERS GET A HONEST, CHEAP ANSWER
   "Sold out" or "still waiting, position 340,201" is returned from
   the edge in milliseconds β€” never a spinning loader that silently
   times out.
# waitingroom/queue_service.py

"""
Virtual Waiting Room: absorbs the thundering herd before it ever
reaches the booking core, and admits requests at a rate the
downstream system has actually signaled it can handle.
"""

from dataclasses import dataclass
from typing import Optional
from datetime import datetime, timedelta
import hashlib


@dataclass
class DownstreamHealth:
    """Live signal from the booking core, refreshed every second."""
    db_latency_ms: float
    payment_success_rate: float
    booking_core_cpu_pct: float


@dataclass
class QueueTicket:
    session_id: str
    bucket: int          # coarse time-bucket for fair shuffling
    shuffle_key: str      # deterministic but unpredictable ordering
    issued_at: datetime


class AdaptiveWaitingRoom:
    """
    Buffers arrivals and admits them to the booking core at a
    rate derived from real downstream health, not a static guess.
    """

    BASE_ADMIT_RATE = 1000      # requests/sec under healthy conditions
    MIN_ADMIT_RATE = 100        # never fully stop; keep some flow moving
    BUCKET_WIDTH_MS = 250

    def __init__(self, queue_store, session_binder, metrics):
        self.queue = queue_store          # ordered structure (e.g. Redis sorted set)
        self.sessions = session_binder    # enforces one active slot per user
        self.metrics = metrics

    async def enqueue(self, session_id: str) -> QueueTicket:
        """
        Called once per user session as they arrive. Idempotent:
        calling twice for the same active session returns the
        SAME ticket, never a second slot.
        """
        existing = await self.sessions.get_active_ticket(session_id)
        if existing:
            return existing

        now = datetime.utcnow()
        bucket = int(now.timestamp() * 1000) // self.BUCKET_WIDTH_MS

        # Shuffle key: hash of session + bucket, NOT arrival timestamp.
        # This defeats "fastest script wins" while staying deterministic.
        shuffle_key = hashlib.sha256(
            f"{session_id}:{bucket}".encode()
        ).hexdigest()

        ticket = QueueTicket(
            session_id=session_id,
            bucket=bucket,
            shuffle_key=shuffle_key,
            issued_at=now,
        )

        await self.queue.add(bucket, shuffle_key, session_id)
        await self.sessions.bind_active_ticket(session_id, ticket)
        self.metrics.increment("queue_enqueued")
        return ticket

    def compute_admit_rate(self, health: DownstreamHealth) -> int:
        """
        Adaptive throttling: shrink the front door the moment the
        booking core shows strain, rather than waiting for it to fail.
        """
        rate = self.BASE_ADMIT_RATE

        if health.db_latency_ms > 50:
            rate *= 0.5
        if health.db_latency_ms > 150:
            rate *= 0.3
        if health.payment_success_rate < 0.98:
            rate *= 0.6
        if health.booking_core_cpu_pct > 80:
            rate *= 0.4

        return max(self.MIN_ADMIT_RATE, int(rate))

    async def run_admission_tick(self, health: DownstreamHealth):
        """
        Runs every ~1 second. Pulls the next batch of tickets, in
        fair shuffled order, and releases them into the booking core.
        """
        admit_count = self.compute_admit_rate(health)
        batch = await self.queue.pop_next(admit_count)

        for session_id in batch:
            await self.sessions.mark_admitted(session_id)

        self.metrics.gauge("admit_rate", admit_count)
        self.metrics.gauge("queue_depth", await self.queue.size())
        return batch

Deep Dive 3: Seat Holds and the Booking Saga (Week 2 & 5 Concepts: Idempotency, Saga Pattern)

You: "Once a request has a real chance at a seat, we hit the exact same trap UPI faces with money: there's a window between 'we've provisionally given you something' and 'you've actually paid for it,' and that window must be bulletproof."

The Challenge:

THE HOLD-THEN-PAY PROBLEM

Between "you've reached the front of the queue and picked seat
24, berth 3A" and "your payment cleared," several seconds to a
few minutes pass β€” Priya fills passenger details, enters an
Aadhaar OTP, chooses UPI, and confirms in her banking app.

During that window:
β€’ The seat CANNOT be given to anyone else (or we oversell)
β€’ The seat CANNOT be held forever (abandoned carts would starve
  real demand β€” remember, 1,999,500 others want this seat)
β€’ If payment fails or times out, the hold must release automatically
β€’ If Priya double-clicks "Pay Now," she must be charged once, not twice
β€’ If the payment gateway is slow to confirm, we must not guess wrong
  in either direction β€” no phantom PNR, no silently lost money

The Solution:

BOOKING STATE MACHINE (A SAGA WITH A TTL)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚                     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                    β”‚
β”‚                     β”‚   ADMITTED   β”‚  (passed the waiting room)         β”‚
β”‚                     β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜                                    β”‚
β”‚                            β”‚ Selects seat                               β”‚
β”‚                            β–Ό                                            β”‚
β”‚                     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                    β”‚
β”‚                     β”‚  SEAT_HELD   β”‚  TTL: 5 minutes                    β”‚
β”‚                     β”‚              β”‚  Held berth is INVISIBLE to the    β”‚
β”‚                     β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜  pool counter β€” it's already gone  β”‚
β”‚           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                           β”‚
β”‚           β”‚ TTL expires    β”‚ Details + OTP  β”‚ Explicitly abandons       β”‚
β”‚           β–Ό                β–Ό                β–Ό                           β”‚
β”‚    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                   β”‚
β”‚    β”‚ HOLD_EXPIRED β”‚ β”‚  PAYMENT_    β”‚ β”‚ HOLD_RELEASEDβ”‚                   β”‚
β”‚    β”‚              β”‚ β”‚  INITIATED   β”‚ β”‚              β”‚                   β”‚
β”‚    β”‚ Seat returnedβ”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Seat returnedβ”‚                   β”‚
β”‚    β”‚ to pool      β”‚        β”‚         β”‚ to pool      β”‚                   β”‚
β”‚    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜        β”‚         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                   β”‚
β”‚                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”                                  β”‚
β”‚              Successβ”‚      β”‚Failure  β”‚Timeout                           β”‚
β”‚                     β–Ό      β–Ό         β–Ό                                  β”‚
β”‚              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                     β”‚
β”‚              β”‚ PNR_   β”‚ β”‚COMPEN- β”‚ β”‚ DEEMED_      β”‚                     β”‚
β”‚              β”‚CONFIRMEDβ”‚ β”‚SATING  β”‚ β”‚ PENDING      β”‚                    β”‚
β”‚              β”‚        β”‚ β”‚RELEASE β”‚ β”‚              β”‚                     β”‚
β”‚              β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Reconcile withβ”‚                    β”‚
β”‚                                     β”‚ gateway; then β”‚                   β”‚
β”‚                                     β”‚ confirm OR    β”‚                   β”‚
β”‚                                     β”‚ refund+releaseβ”‚                   β”‚
β”‚                                     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                    β”‚
β”‚                                                                         β”‚
β”‚  COMPENSATION (the Saga's rollback): if payment fails after the seat    β”‚
β”‚  was held, the seat is atomically returned to the pool counter AND, if  β”‚
β”‚  money was captured, a refund is triggered β€” automatically, no ticket.  β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
# booking/saga_orchestrator.py

"""
Booking Saga Orchestrator.

Coordinates seat hold -> passenger/OTP -> payment -> PNR as one
logical transaction spanning multiple services, with compensating
actions if any step fails. Modeled the same way UPI's debit/credit
saga is: hold first, confirm second, and ALWAYS have a clean way back.
"""

from dataclasses import dataclass
from typing import Optional
from enum import Enum
from datetime import datetime, timedelta
import uuid


class BookingState(Enum):
    ADMITTED = "admitted"
    SEAT_HELD = "seat_held"
    PAYMENT_INITIATED = "payment_initiated"
    PNR_CONFIRMED = "pnr_confirmed"
    HOLD_EXPIRED = "hold_expired"
    HOLD_RELEASED = "hold_released"
    DEEMED_PENDING = "deemed_pending"
    COMPENSATED = "compensated"


@dataclass
class BookingAttempt:
    attempt_id: str
    pool_key: str                 # train:date:class:quota
    state: BookingState
    seat_ref: Optional[str] = None
    payment_ref: Optional[str] = None
    pnr: Optional[str] = None
    hold_expires_at: Optional[datetime] = None
    idempotency_key: str = ""


class BookingSagaOrchestrator:
    """
    Orchestrates the multi-step booking saga for one attempt.

    Guarantees:
    1. IDEMPOTENCY: retrying with the same idempotency_key never
       double-charges or double-allocates a seat.
    2. BOUNDED HOLDS: a held seat is always released within TTL,
       either by success, explicit abandonment, or expiry.
    3. COMPENSATION: any failure after a hold releases the seat and
       refunds any captured payment β€” automatically.
    """

    HOLD_TTL = timedelta(minutes=5)

    def __init__(self, db, seat_pool, payment_gateway, pnr_generator,
                 notifier, expiry_scheduler):
        self.db = db
        self.pool = seat_pool
        self.payments = payment_gateway
        self.pnr_generator = pnr_generator
        self.notifier = notifier
        self.expiry_scheduler = expiry_scheduler

    async def hold_seat(self, pool_key: str, idempotency_key: str) -> BookingAttempt:
        """Step 1: convert an admission ticket into a real, TTL-bound hold."""
        existing = await self.db.get_by_idempotency_key(idempotency_key)
        if existing:
            return existing  # same request retried; return the same attempt

        seat_ref = await self.pool.allocate_from_pool(pool_key)
        if seat_ref is None:
            return BookingAttempt(
                attempt_id=str(uuid.uuid4()),
                pool_key=pool_key,
                state=BookingState.HOLD_EXPIRED,
                idempotency_key=idempotency_key,
            )

        attempt = BookingAttempt(
            attempt_id=str(uuid.uuid4()),
            pool_key=pool_key,
            state=BookingState.SEAT_HELD,
            seat_ref=seat_ref,
            hold_expires_at=datetime.utcnow() + self.HOLD_TTL,
            idempotency_key=idempotency_key,
        )
        await self.db.save(attempt)

        # Schedule an automatic release exactly at TTL β€” never rely on
        # a client callback to free an abandoned seat.
        await self.expiry_scheduler.schedule(
            attempt.attempt_id, run_at=attempt.hold_expires_at
        )
        return attempt

    async def confirm_with_payment(
        self, attempt_id: str, payment_token: str
    ) -> BookingAttempt:
        """Step 2: attempt payment for a held seat, then confirm or compensate."""
        attempt = await self.db.get(attempt_id)

        if attempt.state != BookingState.SEAT_HELD:
            return attempt  # already resolved one way or another

        if datetime.utcnow() > attempt.hold_expires_at:
            await self._release_hold(attempt, reason="EXPIRED_BEFORE_PAYMENT")
            return attempt

        attempt.state = BookingState.PAYMENT_INITIATED
        await self.db.save(attempt)

        result = await self.payments.charge(
            token=payment_token,
            reference=attempt.attempt_id,
            idempotency_key=attempt.idempotency_key,
        )

        if result.status == "SUCCESS":
            pnr = await self.pnr_generator.generate(attempt)
            attempt.pnr = pnr
            attempt.payment_ref = result.reference
            attempt.state = BookingState.PNR_CONFIRMED
            await self.db.save(attempt)
            await self.notifier.send_confirmation(attempt)
            return attempt

        if result.status == "TIMEOUT":
            # Uncertain outcome β€” do NOT guess. Park it for reconciliation,
            # exactly like UPI's "deemed success" state.
            attempt.state = BookingState.DEEMED_PENDING
            await self.db.save(attempt)
            await self.expiry_scheduler.schedule_reconciliation(
                attempt.attempt_id, check_in=timedelta(minutes=10)
            )
            return attempt

        # Clean failure β€” compensate immediately
        await self._release_hold(attempt, reason="PAYMENT_FAILED")
        return attempt

    async def _release_hold(self, attempt: BookingAttempt, reason: str):
        """The Saga's compensating transaction: give the seat back."""
        await self.pool.release_to_pool(attempt.pool_key, attempt.seat_ref)
        attempt.state = BookingState.HOLD_RELEASED
        await self.db.save(attempt)

        if attempt.payment_ref:
            await self.payments.refund(attempt.payment_ref)

    async def reconcile_deemed_pending(self, attempt_id: str):
        """
        Called by the background reconciliation worker for any
        booking stuck in DEEMED_PENDING β€” resolves it one way,
        permanently, based on the payment gateway's true record.
        """
        attempt = await self.db.get(attempt_id)
        actual = await self.payments.check_status(attempt.attempt_id)

        if actual.status == "SUCCESS":
            pnr = await self.pnr_generator.generate(attempt)
            attempt.pnr = pnr
            attempt.state = BookingState.PNR_CONFIRMED
            await self.db.save(attempt)
            await self.notifier.send_confirmation(attempt)
        else:
            await self._release_hold(attempt, reason="RECONCILED_AS_FAILED")

Deep Dive 4: Aadhaar, OTP, and the War on Bots (Week 9 Concepts: Security Architecture)

You: "Every seat a bot or a bulk-booking agent grabs in the first ten seconds is a seat a genuine passenger never sees. Since mid-2025, Indian Railways has treated this as a core architecture problem, not just a policy problem."

The Challenge:

IRCTC'S REAL FRAUD SCALE (2025 DISCLOSED FIGURES)

3.03 crore (30.3 million) suspicious user IDs deactivated
6.05 crore (60.5 million) IDs placed under revalidation
13,343 suspicious email domains blocked
4.18 lakh (418,000) suspicious PNRs flagged
501 complaints filed on the National Cyber Crime Portal

The threat isn't "hackers breaking encryption." It's:
β”œβ”€β”€ Scripts that auto-fill and auto-submit faster than any human
β”œβ”€β”€ Agents/touts who book Tatkal tickets in bulk to resell at a premium
β”œβ”€β”€ Farms of throwaway accounts + disposable emails to dodge per-user
β”‚   booking limits
└── Data-center IPs (not real home/mobile networks) hammering the
    booking API directly, bypassing the normal app/website entirely

The Solution:

LAYERED DEFENSE, TIMED TO THE TATKAL WINDOW

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚  LAYER 1: IDENTITY VERIFICATION (regulatory, since mid-2025)            β”‚
β”‚  ──────────────────────────────────────────────────────────             β”‚
β”‚  β€’ Aadhaar authentication required for Tatkal on web/app                β”‚
β”‚    (effective 1 July 2025)                                              β”‚
β”‚  β€’ Aadhaar-based OTP mandatory for Tatkal β€” web, app, AND counter/      β”‚
β”‚    agent bookings (effective 15 July 2025)                              β”‚
β”‚  β€’ Raises the cost of running throwaway accounts sharply                β”‚
β”‚                                                                         β”‚
β”‚  LAYER 2: TIME-WINDOW RESTRICTIONS                                      β”‚
β”‚  ───────────────────────────────────                                    β”‚
β”‚  β€’ Authorized agents blocked from booking in the first 30 minutes       β”‚
β”‚    of the Tatkal window (AC: 10:00-10:30, non-AC: 11:00-11:30)          β”‚
β”‚  β€’ This protects the highest-value first seconds for individual         β”‚
β”‚    travelers, while still letting agents serve customers afterward      β”‚
β”‚                                                                         β”‚
β”‚  LAYER 3: SESSION & DEVICE BINDING                                      β”‚
β”‚  ───────────────────────────────────                                    β”‚
β”‚  β€’ One active login per user, one Tatkal booking per opening window     β”‚
β”‚  β€’ Device fingerprint bound to the session; re-login invalidates old    β”‚
β”‚    sessions rather than allowing parallel sessions                      β”‚
β”‚                                                                         β”‚
β”‚  LAYER 4: BEHAVIORAL & VELOCITY SIGNALS                                 β”‚
β”‚  ─────────────────────────────────────────                              β”‚
β”‚  β€’ Fill-speed analysis: real humans take measurable time to type        β”‚
β”‚    passenger names and read OTP screens; scripts don't                  β”‚
β”‚  β€’ Request velocity per account, per device, and per IP/ASN             β”‚
β”‚  β€’ Data-center / hosting-provider IP ranges scored higher risk than     β”‚
β”‚    residential or mobile carrier IPs                                    β”‚
β”‚  β€’ Repeated failed-then-retried patterns across many "different"        β”‚
β”‚    accounts sharing a device fingerprint or payment instrument          β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
# security/tatkal_risk_gate.py

"""
Risk gate specific to the Tatkal booking window. Runs BEFORE the
booking saga is allowed to hold a seat, so that flagged traffic
never even competes for real inventory.
"""

from dataclasses import dataclass
from typing import List, Tuple
from datetime import datetime, time


@dataclass
class TatkalRiskSignals:
    user_id: str
    device_fingerprint: str
    is_authorized_agent: bool
    aadhaar_verified: bool
    otp_verified: bool

    # Velocity
    requests_per_min_this_device: int
    accounts_seen_on_this_device_today: int

    # Network
    is_datacenter_ip: bool
    ip_request_rate_per_sec: float

    # Behavioral
    form_fill_time_ms: int          # time between page load and submit
    tatkal_bookings_today: int      # this user, this opening window


class TatkalRiskGate:
    """
    Fast, explainable rules run first (cheap, deterministic);
    an ML model catches subtler patterns the rules miss.
    Target decision latency: < 50ms, same discipline as a real-time
    payment fraud check.
    """

    MIN_HUMAN_FILL_TIME_MS = 1200  # a script can submit in <100ms

    def __init__(self, ml_model, agent_window_config):
        self.model = ml_model
        self.agent_window = agent_window_config  # (start, end) per class

    async def evaluate(
        self, signals: TatkalRiskSignals, now: datetime, travel_class: str
    ) -> Tuple[str, List[str]]:
        """
        Returns: (decision, triggered_rules)
        decision: "ALLOW", "BLOCK", "STEP_UP" (extra OTP / CAPTCHA)
        """
        triggered: List[str] = []

        # RULE 1: Identity verification is mandatory, not optional
        if not (signals.aadhaar_verified and signals.otp_verified):
            return "BLOCK", ["IDENTITY_NOT_VERIFIED"]

        # RULE 2: Agent time-window restriction
        if signals.is_authorized_agent:
            start, end = self.agent_window.get(travel_class)
            if start <= now.time() <= end:
                return "BLOCK", ["AGENT_WINDOW_RESTRICTED"]

        # RULE 3: One Tatkal booking per user per window
        if signals.tatkal_bookings_today > 0:
            return "BLOCK", ["ALREADY_BOOKED_THIS_WINDOW"]

        # RULE 4: Device fan-out β€” many accounts, one device, same morning
        if signals.accounts_seen_on_this_device_today > 3:
            triggered.append("DEVICE_ACCOUNT_FANOUT")

        # RULE 5: Network origin
        if signals.is_datacenter_ip:
            triggered.append("DATACENTER_IP")
        if signals.ip_request_rate_per_sec > 5:
            triggered.append("HIGH_IP_VELOCITY")

        # RULE 6: Inhuman speed
        if signals.form_fill_time_ms < self.MIN_HUMAN_FILL_TIME_MS:
            triggered.append("SUBHUMAN_FILL_SPEED")

        # ML model catches the combinations rules can't enumerate
        ml_score = await self.model.predict(signals)

        if ml_score > 0.9 or len(triggered) >= 3:
            return "BLOCK", triggered + ["ML_HIGH_RISK"]

        if ml_score > 0.6 or len(triggered) >= 1:
            return "STEP_UP", triggered

        return "ALLOW", triggered

Phase 5: Scaling and Edge Cases (5 minutes)

Interviewer: "Walk me through your Tatkal-day playbook β€” what actually happens in the hours around 10 AM?"

Scaling Strategies

THE TATKAL-DAY PLAYBOOK

T-30 minutes (09:30 AM):
β”œβ”€β”€ Pre-scale booking-core compute for known hot trains (historical
β”‚   demand data predicts which routes will spike)
β”œβ”€β”€ Warm caches for search/availability on those routes
β”œβ”€β”€ Pre-authenticate sessions: users who logged in early get their
β”‚   Aadhaar OTP validated ahead of time where policy allows, so the
β”‚   critical path at 10:00:00 is shorter

T-5 minutes (09:55 AM):
β”œβ”€β”€ Freeze non-critical deployments and config changes
β”œβ”€β”€ Initialize hot-pool counters (Deep Dive 1) for every train
β”‚   opening Tatkal quota this run
β”œβ”€β”€ Waiting room admission starts accepting connections, holding
β”‚   everyone at the door

T-0 (10:00:00 AM):
β”œβ”€β”€ Waiting room begins admitting at the adaptive rate (Deep Dive 2)
β”œβ”€β”€ Search/enquiry traffic is served entirely from the read plane β€”
β”‚   it never touches booking-core capacity
β”œβ”€β”€ Booking core only ever sees traffic proportional to real
β”‚   inventory, not raw demand

T+15 minutes (10:15 AM):
β”œβ”€β”€ Hot pools for popular trains are exhausted; counters read zero
β”œβ”€β”€ Waiting room shifts remaining queued users straight to "sold out"
β”œβ”€β”€ Auto-scale booking-core compute back down; read plane keeps
β”‚   absorbing the long tail of enquiries

ONGOING, ALL DAY:
β”œβ”€β”€ Waitlist promotion runs as cancellations arrive, walking the
β”‚   waitlist_queue in strict order (WL β†’ RAC β†’ CNF)
β”œβ”€β”€ Chart preparation (~4 hours before departure) finalizes the
β”‚   passenger list and runs one last promotion pass

Edge Cases

EDGE CASE 1: Payment succeeds after the hold already expired

Problem: Priya's payment gateway call is slow; her 5-minute hold
expires and the seat is reassigned, but her bank confirms payment
30 seconds later.

Solution:
β”œβ”€β”€ Hold expiry does NOT cancel an in-flight payment call β€” it only
β”‚   stops new holds and marks this attempt for reconciliation
β”œβ”€β”€ Reconciliation worker checks true payment status; if charged and
β”‚   the seat is gone, auto-refund with priority, plus a clear
β”‚   notification β€” never a silent loss

EDGE CASE 2: Double-submit from an impatient user

Problem: The confirm button is tapped twice within a second because
the UI felt slow.

Solution:
β”œβ”€β”€ Client generates one idempotency_key per booking attempt
β”œβ”€β”€ Every downstream call (hold, payment, PNR) is keyed on it
β”œβ”€β”€ Second submission returns the SAME result as the first β€”
β”‚   no second charge, no second seat consumed

EDGE CASE 3: Last-minute cancellation triggers a waitlist promotion race

Problem: A confirmed passenger cancels 3 hours before departure.
Multiple waitlisted passengers could be considered for that one seat.

Solution:
β”œβ”€β”€ Promotion is a single-writer, ordered walk over waitlist_queue β€”
β”‚   never a parallel "first request wins" race
β”œβ”€β”€ Promoted passengers are notified via SMS before the seat is
β”‚   irreversibly reassigned to someone further down the list

EDGE CASE 4: The write plane and read plane accidentally share load

Problem (real, documented pattern): if search/enquiry traffic and
booking traffic share infrastructure, a spike in one degrades the
other. IRCTC's website has visibly gone down or been throttled
during Tatkal hours on multiple occasions (e.g. December 2024,
users saw "e-ticketing service will not be available for the next
one hour" precisely during the Tatkal window).

Lesson learned:
β”œβ”€β”€ Enquiry and booking MUST be physically separate service tiers,
β”‚   not just logically separate endpoints on shared infrastructure
β”œβ”€β”€ Rate limit the enquiry path aggressively β€” it's read-only and
β”‚   idempotent, so aggressive throttling costs nothing but a stale
β”‚   number on screen
β”œβ”€β”€ Give the booking path its own dedicated, pre-scaled capacity
β”‚   that enquiry traffic can never consume

EDGE CASE 5: A single hot train's chaos leaking into other trains

Problem: If seat inventory for all trains lives in one shared
database cluster, one popular train's 280,000 req/sec burst can
starve query capacity for the other 12,999 trains running that day.

Solution:
β”œβ”€β”€ Hot pools are isolated per (train, date) β€” a busy Rajdhani's
β”‚   admission gate runs on infrastructure that a quiet local
β”‚   passenger train's booking never touches
β”œβ”€β”€ The Seat Inventory Service intentionally does NOT share a
β”‚   connection pool or lock table across unrelated trains

Phase 6: Monitoring and Operations

You: "For a system where 'temporarily down' during Tatkal means real families can't travel, observability during the opening minutes matters more than almost anything else we've built."

Monitoring Dashboard

TATKAL WINDOW β€” LIVE OPERATIONS DASHBOARD (10:00:00 - 10:15:00)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚  ADMISSION FUNNEL (last 60 seconds, this train)                         β”‚
β”‚  ────────────────────────────────────────────────                       β”‚
β”‚  Requests received        β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  1,998,412               β”‚
β”‚  Admitted to booking core  β–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘     14,207              β”‚
β”‚  Seats successfully held   β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘        512              β”‚
β”‚  PNRs confirmed             β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘        487             β”‚
β”‚                                                                         β”‚
β”‚  ─────────────────────────────────────────────────────────────────────  β”‚
β”‚                                                                         β”‚
β”‚  WRITE PLANE HEALTH                    READ PLANE HEALTH                β”‚
β”‚  ───────────────────                   ─────────────────                β”‚
β”‚  DB latency (p99):     38 ms            Cache hit rate:      99.4%      β”‚
β”‚  Payment success:      98.7%            Enquiry latency p99: 60 ms      β”‚
β”‚  Admit rate (current):  620/sec         Enquiry rate:        6,700/sec  β”‚
β”‚                                                                         β”‚
β”‚  ─────────────────────────────────────────────────────────────────────  β”‚
β”‚                                                                         β”‚
β”‚  FRAUD GATE (this train, this window)                                   β”‚
β”‚  ─────────────────────────────────────                                  β”‚
β”‚  Blocked β€” identity not verified:        41,203                         β”‚
β”‚  Blocked β€” agent window restriction:      2,891                         β”‚
β”‚  Blocked β€” device/IP velocity rules:     18,447                         β”‚
β”‚  Step-up (extra verification) triggered:  6,120                         β”‚
β”‚                                                                         β”‚
β”‚  ─────────────────────────────────────────────────────────────────────  β”‚
β”‚                                                                         β”‚
β”‚  ALERTS (last 15 minutes)                                               β”‚
β”‚  ────────────────────────                                               β”‚
β”‚  🟒 10:00:00 - Waiting room opened, admitting at adaptive rate          β”‚
β”‚  🟑 10:02:14 - DB latency crossed 40ms, admit rate throttled 20%        β”‚
β”‚  🟒 10:04:50 - Hot pool for train 12301 exhausted (sold out)            β”‚
β”‚  🟒 10:15:00 - Booking core auto-scaled down, read plane unaffected     β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

SLOs for the Tatkal System

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚  SLO 1: ZERO OVERSELL                                                   β”‚
β”‚  ──────────────────────                                                 β”‚
β”‚  Target: 100% β€” the pool counter and the durable allocator never        β”‚
β”‚  disagree on how many seats have been given out                         β”‚
β”‚  Error budget: 0. This is non-negotiable, exactly like UPI's money      β”‚
β”‚  safety guarantee.                                                      β”‚
β”‚                                                                         β”‚
β”‚  SLO 2: ADMISSION DECISION LATENCY                                      β”‚
β”‚  ───────────────────────────────────                                    β”‚
β”‚  Target: p99 < 200ms for "you're in the queue" or "sold out"            β”‚
β”‚  Current: p99 = 140ms                                                   β”‚
β”‚                                                                         β”‚
β”‚  SLO 3: READ/WRITE PLANE ISOLATION                                      β”‚
β”‚  ────────────────────────────────────                                   β”‚
β”‚  Target: enquiry traffic causes zero measurable increase in booking     β”‚
β”‚  path latency, even at 4 lakh enquiries/minute                          β”‚
β”‚                                                                         β”‚
β”‚  SLO 4: PAYMENT-TO-SEAT INTEGRITY                                       β”‚
β”‚  ──────────────────────────────────                                     β”‚
β”‚  Target: 100% of captured payments resolve to either a confirmed        β”‚
β”‚  PNR or a completed refund within 30 minutes                            β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Interview Conclusion

Interviewer: "Solid. A few rapid-fire questions."

Interviewer: "Why not just add more database servers? Isn't this a capacity problem?"

You: "Only 500 seats exist. No amount of extra capacity changes that fact β€” it only changes how fast you can tell 1,999,500 people 'no.' The real engineering problem is admission control: reject the impossible cases as early and cheaply as possible, and only let real contention reach real inventory."

Interviewer: "Why is search/availability a completely separate system from booking?"

You: "Because their failure modes are opposite. Search can be stale by a second and nobody's hurt β€” a cache serving slightly outdated availability is a fine trade-off. Booking cannot be stale by even a millisecond, or two people get sold the same berth. Mixing them means the system that can tolerate staleness ends up dictating the availability of the system that cannot."

Interviewer: "What's the single most important design decision here?"

You: "Making the hot pool counter dumb on purpose. It does one atomic operation and nothing else β€” no joins, no passenger data, no payment logic. Every extra responsibility you bolt onto a hot key is extra work multiplied by 280,000 requests per second. The durable, feature-rich logic only runs for the ~500 requests that actually earned the right to reach it."

Interviewer: "If you were improving today's real IRCTC system, what would you change?"

You: "Based on documented outages during Tatkal hours: stronger physical separation between the enquiry and booking tiers, so a search traffic spike can never again produce a 'service unavailable for the next hour' message during the exact minute people need booking most. I'd also push the identity-verification and OTP steps to happen before the queue, not during it, so verified users don't burn their queue position on an avoidable extra round-trip."


Summary: Concepts Applied from 10-Week Course

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                                                                         β”‚
β”‚         CONCEPTS FROM 10-WEEK COURSE IN IRCTC TATKAL DESIGN             β”‚
β”‚                                                                         β”‚
β”‚  WEEK 1: DATA AT SCALE                                                  β”‚
β”‚  β”œβ”€β”€ Hot keys & skew: the 500-seat pool as the system's hottest row     β”‚
β”‚  β”œβ”€β”€ Rate limiting: per-user, per-device, per-IP admission limits       β”‚
β”‚  └── Session store design: one active session/queue slot per user       β”‚
β”‚                                                                         β”‚
β”‚  WEEK 2: FAILURE-FIRST DESIGN                                           β”‚
β”‚  β”œβ”€β”€ Idempotency: idempotency_key across hold, payment, and PNR         β”‚
β”‚  β”œβ”€β”€ Timeouts: bounded seat holds (5-minute TTL), bounded payment       β”‚
β”‚  β”‚   waits before falling back to reconciliation                        β”‚
β”‚  └── The flash-sale pattern: claim inventory, then checkout, mirrors    β”‚
β”‚      exactly what Week 2's capstone (flash sale for a retailer) covers  β”‚
β”‚                                                                         β”‚
β”‚  WEEK 3: MESSAGING & ASYNC PROCESSING                                   β”‚
β”‚  β”œβ”€β”€ Backpressure & flow control: the waiting room's adaptive admit     β”‚
β”‚      rate, driven by live downstream signals                            β”‚
β”‚  └── Queue vs stream: ordered waitlist_queue for fair promotion         β”‚
β”‚                                                                         β”‚
β”‚  WEEK 4: CACHING                                                        β”‚
β”‚  β”œβ”€β”€ Thundering herd mitigation: the entire waiting-room design exists  β”‚
β”‚      to prevent one                                                     β”‚
β”‚  └── Cache invalidation: availability snapshots refreshed on a short    β”‚
β”‚      TTL, decoupled from the authoritative pool counter                 β”‚
β”‚                                                                         β”‚
β”‚  WEEK 5: CONSISTENCY & COORDINATION                                     β”‚
β”‚  β”œβ”€β”€ Saga pattern: hold β†’ pay β†’ confirm, with compensation on failure   β”‚
β”‚  β”œβ”€β”€ Single-writer serialization: durable allocation for the reduced,   β”‚
β”‚      post-admission-gate volume                                         β”‚
β”‚  └── Leader-election-style ownership: one authoritative writer per pool β”‚
β”‚                                                                         β”‚
β”‚  WEEK 9: MULTI-TENANCY & SECURITY                                       β”‚
β”‚  β”œβ”€β”€ Security architecture: layered identity, time-window, and          β”‚
β”‚      behavioral defenses against bots and bulk-booking agents           β”‚
β”‚  └── Real regulatory response: Aadhaar OTP mandate, agent restrictions  β”‚
β”‚                                                                         β”‚
β”‚  WEEK 10: PRODUCTION READINESS                                          β”‚
β”‚  β”œβ”€β”€ SLOs: zero-oversell, admission latency, plane isolation targets    β”‚
β”‚  β”œβ”€β”€ Capacity planning: the Tatkal-day playbook, pre-scale/post-scale   β”‚
β”‚  └── Incident learnings: documented real-world outages as case studies  β”‚
β”‚                                                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Why IRCTC's Tatkal System Matters

╔═════════════════════════════════════════════════════════════════════════╗
β•‘                                                                         β•‘
β•‘             WHY THIS IS A DIFFERENT KIND OF HARD PROBLEM                β•‘
β•‘                                                                         β•‘
β•‘  SCALE                                                                  β•‘
β•‘  ─────                                                                  β•‘
β•‘  β€’ Up to 2 million attempts against 500 seats on a single hot train     β•‘
β•‘  β€’ 37,410 tickets/minute nationally at record peak                      β•‘
β•‘  β€’ 300,000+ concurrent users sustained during Tatkal windows            β•‘
β•‘                                                                         β•‘
β•‘  A DIFFERENT SHAPE OF HARD                                              β•‘
β•‘  ──────────────────────────                                             β•‘
β•‘  β€’ UPI's hard problem is moving money correctly between many parties    β•‘
β•‘  β€’ Netflix's hard problem is delivering bytes efficiently at scale      β•‘
β•‘  β€’ IRCTC's Tatkal problem is admission control: making 99.975% of       β•‘
β•‘    all traffic disappear cheaply, fairly, and in milliseconds, so       β•‘
β•‘    the remaining 0.025% can be served perfectly                         β•‘
β•‘                                                                         β•‘
β•‘  HUMAN STAKES                                                           β•‘
β•‘  ────────────                                                           β•‘
β•‘  β€’ A failed booking can mean a missed job interview, a missed           β•‘
β•‘    funeral, a missed wedding β€” Tatkal exists precisely for urgent,      β•‘
β•‘    last-minute travel                                                   β•‘
β•‘  β€’ A double-booked seat means two real families in conflict at a        β•‘
β•‘    real train door                                                      β•‘
β•‘                                                                         β•‘
β•‘  ONGOING EVOLUTION                                                      β•‘
β•‘  ──────────────────                                                     β•‘
β•‘  β€’ Aadhaar-OTP authentication (2025) reshaped identity verification     β•‘
β•‘  β€’ Agent booking windows (2025) reshaped fairness policy                β•‘
β•‘  β€’ Modernization targets aim for 1.5 lakh tickets/minute capacity,      β•‘
β•‘    up from ~32,000/minute today                                         β•‘
β•‘                                                                         β•‘
β•‘  ════════════════════════════════════════════════════════════════════   β•‘
β•‘                                                                         β•‘
β•‘  "The hardest part of this system was never the database. It was        β•‘
β•‘   accepting that almost every request must fail, and designing          β•‘
β•‘   for that failure to be instant, honest, and fair."                    β•‘
β•‘                                                                         β•‘
β•šβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•

Self-Assessment Checklist

After studying this case study, you should be able to:

Architecture:

  • Explain why search/enquiry and booking must be architecturally separate planes
  • Design a hot-key admission gate that protects a durable database from extreme contention
  • Trace the full booking flow from queue admission to PNR confirmation

Distributed Systems:

  • Implement an adaptive virtual waiting room driven by live downstream health signals
  • Design a Saga-based hold-then-pay flow with automatic compensation
  • Explain why single-writer serialization can outperform naive horizontal sharding when the total inventory is small

Scale:

  • Calculate contention ratios and peak instantaneous request rates for flash-demand events
  • Explain why storage is rarely the bottleneck in extreme-contention systems
  • Design a Tatkal-day capacity playbook (pre-scale, admit, exhaust, de-scale)

Security:

  • Design layered anti-bot defenses: identity verification, time-windows, velocity, behavior
  • Explain the real-world rationale behind Aadhaar OTP and agent-window restrictions

Operations:

  • Define SLOs appropriate for a zero-oversell, admission-control-dominated system
  • Monitor an admission funnel end-to-end, from raw requests to confirmed PNRs
  • Learn from real documented outages where read and write traffic shared infrastructure

Sources

Statistics and Scale:

Architecture and Technical Design:

Policy and Regulatory Changes (2025):

Real Incidents (Outages During Tatkal Hours):


Further Reading

Official Sources:

  • IRCTC Official Website: https://www.irctc.co.in/
  • Indian Railways / Ministry of Railways Press Releases: https://www.pib.gov.in/ (search "Tatkal")
  • CRIS (Centre for Railway Information Systems): the organization that builds and operates PRS and IRCTC's backend ticketing platforms

Related Systems to Study:

  • Ticketmaster / BookMyShow β€” concert and event ticketing at flash-sale scale: nearly identical admission-control and hot-inventory problems
  • Amazon Prime Day / e-commerce flash sales: this course's own Week 2 capstone ("Flash Sale for MegaMart") covers the same claim-then-checkout pattern
  • Airline GDS systems (Amadeus, Sabre): seat inventory allocation at a different but related scale
  • UPI (Bonus Problem 1 in this series): the closest sibling problem β€” atomic, reversible, hold-then-confirm transactions under extreme concurrency

Foundational Concepts (this course):

  • Week 1, Day 4 β€” Hot Keys and Skew
  • Week 2, Capstone β€” Flash Sale for MegaMart
  • Week 3, Day 3 β€” Backpressure and Flow Control
  • Week 4, Day 3 β€” Thundering Herd
  • Week 5, Day 2 β€” Distributed Transactions: The Saga Pattern

End of Bonus Problem 11: IRCTC Tatkal Ticket Booking System

"Two million people, five hundred seats, ten seconds. The system that handles this gracefully isn't the one with the biggest database β€” it's the one that knows, instantly and honestly, who to turn away."

πŸ’¬ Public Discussion: Comments are visible to all users. Please be respectful and mindful of what you share.

Discussion (0)

Sort by:

Sign in to join the discussion