War stories.
What really happened when real systems hit real limits, in two minutes each, with the sources. Then take the page yourself: incidents played out in the channel, modelled on real postmortems, where you make the calls.
20 real stories · 5 war rooms
war room · you're on callTake the page
Fictional company and people; each night is modelled on a real public postmortem, named at the end.- SEV-1 · 19:15 · Prism Edge+60 xpEvery website behind us is down at oncea CDN and web firewall in front of 40,000 websites~12 min · 5 calls
- SEV-1 · 23:07 · Tijori Cloud+60 xpOne typo takes out a storage regionan object-storage service whose Mumbai region holds files for 30,000 apps~13 min · 5 calls
- SEV-1 · 11:42 · Sangam Payments Exchange+60 xpStatus checks flood the payment switcha real-time payment switch between 300 banks and 50 payment apps~12 min · 5 calls
- SEV-1 · 18:10 · Khelverse+60 xpThe address book that won't heala game platform with 9 million players a day~14 min · 5 calls
- SEV-1 · 05:04 · Shakha+60 xpThe wipe ran on the primarya code-hosting service for 1.5 million developers~13 min · 5 calls
real stories · two minutes eachIt really happened
- Money · 37signals2 min37signals left the cloud and cut its yearly bill from $3.2 million to $1.3 millionThe makers of Basecamp and HEY bought about $700,000 of servers and, by their own math, got it back within a year.2022–2024
- Money · Amazon Web Services2 minA brand-new S3 test bucket ran up a $1,300 bill in a dayA developer made a storage bucket for a quick test. Strangers' machines sent it nearly 100 million requests, and he got the bill.2024
- Live migration · Netflix2 minA corrupted database pushed Netflix into a seven-year move to the cloudIn 2008 Netflix couldn't ship DVDs for three days. Its answer took seven years and rebuilt almost all of its technology.2008–2016
- Money · Amazon Prime Video2 minA Prime Video team cut a tool's cost by 90% by turning it into a monolithNot all of Prime Video, just one monitoring tool, whose pieces turned out far cheaper living together in one program.2023
- People · GitLab2 minAn engineer wiped GitLab's main database, and the backups weren't thereAn engineer wiped what they thought was the copy. It was the original, and none of GitLab's backup methods came to the rescue.2017
- Scale surprise · CoWIN (Government of India)2 minCoWIN took 383 million API hits in three hours, and still had no slots to giveWhen vaccine sign-ups opened to every adult in India, the servers held up. The vaccines were the bottleneck.2021
- Live migration · Discord2 minDiscord moved trillions of messages to a new database in nine daysNot overnight, and not in a panic. Discord planned the move for years, then copied trillions of messages in nine days.2017–2022
- Live migration · Dropbox2 minDropbox moved its users' files out of Amazon's cloud and onto its ownDropbox built its own storage system, then gave itself six months to grow it past 500 petabytes.2013–2016
- People · Flipkart2 minFlipkart's founders said sorry the day after the Big Billion DayFlipkart planned for 20x traffic growth and still fell over. The next day its founders told customers, "Yesterday, we failed that trust."2014
- Strange but true · Google Cloud2 minFour lightning strikes cost Google Cloud a sliver of its customers' dataLightning hit the power grid near a Google data center four times in a row. Most data survived; 0.000001% of it did not.2015
- Outage · National Stock Exchange of India (NSE)2 minIndia's biggest stock exchange stopped trading for four hoursNSE's trading engine was fine. The risk system next to it wasn't, and without it the market could not stay open.2021
- People · Swiggy2 minInside Swiggy's New Year's Eve war room400 people in one Bengaluru office, 950 in one Slack channel and 3 lakh delivery partners behind them, all for Swiggy's busiest night.2022
- Strange but true · NASA (Jet Propulsion Laboratory and Lockheed Martin Astronautics)2 minNASA lost a Mars orbiter because one file used the wrong unitsOne program wrote pound-seconds. The next one read newton-seconds. Nine months later the spacecraft arrived at Mars about 170 km too low.1998–1999
- Outage · Fastly2 minOne customer's valid settings change broke 85% of Fastly's networkThe bug shipped on 12 May and sat quiet for almost four weeks. On 8 June, one customer's ordinary change woke it up.2021
- Outage · Cloudflare2 minOne line of regex knocked Cloudflare offline for 27 minutesA new firewall rule, meant only to watch traffic, maxed out the CPUs serving web traffic across Cloudflare's whole network.2019
- Outage · Amazon Web Services2 minOne wrong input took down Amazon S3 in Virginia for about four hoursAn engineer meant to remove a few servers. One input was wrong, and S3 in Northern Virginia stopped answering for hours.2017
- Scale surprise · Niantic and Google Cloud2 minPokémon GO got 50 times the traffic it planned forThe team planned for a worst case of 5x their traffic target. Launch brought 50x, ten times that worst case.2016
- Outage · Roblox2 minRoblox was down for 73 hours because its service directory chokedFifty million people play Roblox every day. For three days they couldn't, and the cause hid in the tool that coordinates everything.2021
- Strange but true · Microsoft (Windows Azure)2 minWindows Azure tripped over 29 FebruaryTo get "one year from today", the code added 1 to the year. On 29 February 2012 that produced a date that doesn't exist.2012
- Scale surprise · Zoom2 minZoom went from 10 million to 300 million daily meeting participantsZoom's 2020 growth was real. Its "300 million users" wasn't. It counted meeting participants, so one person could count many times.2019–2020