Twilio's balances went read-only. The cards kept getting charged.

transcript

Your card gets charged. Then again. And again. Because Twilio's billing couldn't write down that you'd paid.

1:35 a.m. Twilio keeps live balances in Redis. The network drops. When it's back, every copy asks the master for a full resync. At once. The master buckles. By 2:39, services are failing.

2:42. On-call restarts Redis on the master. But the config file is wrong. It recovers from a file that doesn't exist. Every balance, gone. And it boots as a slave of itself. Read-only.

Now billing sees a low balance. So it charges your card. It tries to save the new balance. Read-only. Nothing changes. Balance still low. So it charges you again. Some cards got deactivated. Then those accounts got suspended.

3:28. Monitoring spots the anomaly. 4:10. Billing is switched off. The charging stops. They rebuild the balances and turn it back on. It happens again.

1 point 4 percent of Twilio's customers were hit. Another 0 point 3 percent when billing came back too soon. Refunds, plus a 10 percent credit on a month of spend.

Twilio began adding fail-safes. No charges or suspensions if balances can't be written. If you can't record that you charged someone, don't charge them again.

Full story in the article.