The failover worked perfectly. That's what broke it.

transcript

Their failover worked perfectly. That's what broke everything. This is GOV.UK Notify, late 2019.

Notify sends texts for the UK government. Flood warnings. Sign-in codes. It had two SMS providers. All traffic went through one. A single error from it? Swap everything to the other. Automatically.

Then, out of nowhere, a large spike of texts hits provider A. A can't handle the load. It starts to fail. Failover kicks in, exactly as designed. Everything moves to B. And B starts returning errors too.

Here's the catch. The team's own account? The providers most likely needed time to scale up. The whole spike landed at once, on a provider that hadn't seen it coming.

The fix: always split. Each text picks a provider at random. About fifty-fifty. An error shifts just 10 points. At most once a minute. Quiet for an hour? Both drift 10 points back.

Keep both warm. Never move traffic onto a provider that's had none.

Full breakdown in the lecture.