Diwali, 7:40 pm: 4% of tills can't sync

transcript

Diwali, 7:40 pm. A deploy went out 20 minutes ago. Now 4 percent of tills can't sync a single bill.

7:20. A backend deploy goes out. 7:40. Devices on app 5 point 2 get HTTP 400. Every sync. Bills pile up offline. Merchants call support. The interviewer asks: is anything actually lost? Not yet.

Offline-first. Each device holds 72 hours of bills. So support's first line: your bills are safe. Keep billing.

Why not push a fixed app? That's new risk, mid-Diwali dinner. And some devices take months to update. Roll back instead. It's a change you already know. Cell by cell, worst first. A rollback is a deploy too.

But the rollback brings a second wave. 24,000 devices reconnect. About 40 bills each. A million bills, all at once. So ingest answers retry after, with jitter. Spread over 10 to 15 minutes.

Then the postmortem. Why didn't the canary catch it? Likely cause: the canary cell had no devices on 5 point 2. And an overall average hides a 4 percent segment.

Pick canary cells with every active app version. And roll back automatically per app version, not on the average.

Full breakdown in the lecture.