TSB moved 5.2 million customers to a new platform, and their sessions landed in the wrong data centre

transcript

You log in to your bank. One click later. Your session has expired. Now imagine that across 5 point 2 million customers.

TSB, a UK bank, moves to a brand new platform. One weekend. 22 April 2018. The new system goes live. Almost at once, online, mobile and phone banking fail. All 550 branches hit. Back to business as usual? Not until December.

But here's the catch. The data itself migrated successfully. One big problem lived between two data centres.

Two data centres, both live. So either could fail without an outage. Your login lands in data centre one. Your next click? Sometimes data centre two. It's never heard of you. Session expired. In week one, digital banking was at times completely down.

The twist? A test might have caught some of it. The plan: test both data centres, live together. Weeks before go-live, TSB agreed to test only one. Informal. Not documented. Not escalated.

TSB paid 32 point 7 million pounds to customers. Then regulators fined it 48 point 65 million pounds.

Production had two data centres. The test had one. Test the system the way it will really run.

Full story in the article.