Twilio's billing system kept "in-flight" account balances in Redis, an in-memory database: one master and several copies, called slaves, across data centres. Auto-recharge tops up a low balance from the customer's card.
At 1:35 a.m. PDT on 18 July 2013, the network between the copies and the master dropped. When it came back, every copy asked the master for a full resync at the same time. The master buckled. By 2:39 a.m. services that depended on it were failing.
At 2:42, on-call engineers restarted Redis on the master. Twilio calls this a misdiagnosis, and an incorrect configuration file made it worse twice over. Redis tried to recover from a log file that didn't exist instead of its snapshot, so it came up with no balance data. And it booted as a slave of itself, which made it read-only.
Now the billing system saw balances it couldn't update. So it did what it was built to do: charge cards to top them up. The balance never changed, so it charged them again. Some cards were deactivated by the repeated charges, and the next failed recharge suspended those accounts.
Monitoring flagged the anomaly at 3:28 a.m., and billing was switched off at 4:10 a.m. Twilio rebuilt the balances from its second set of books in another database, but when billing came back at 11:58 a.m. the anomaly returned, so it went off again. Twilio refunded the charges, gave affected accounts a 10% credit on their last 30 days of spend, and restored billing gradually, with the all-clear on 19 July.
The fixes: Redis restarts on the master were disabled, and Twilio said it was adding fail-safes so billing won't charge cards or suspend accounts when balances can't be written.
- 1.4% of Twilio's customers affected; 1.1% by wrong charges or suspensions at first
- Another 0.3% hit when billing came back too soon
- Restart at 2:42 a.m., alert at 3:28, billing off at 4:10
- A 10% credit on the last 30 days of spend
Why it matters: When a system can't record what it did, it must stop, not try again. Make money-moving code fail safe.