2:58:31: the primary dies after confirming a trade

transcript

2:58:31. Expiry afternoon. The primary matching server loses power. It had just confirmed a trade.

The primary had reached message 812,330,114, and sent the trade. Its hot standby had only reached 109. Five messages behind. Members saw a trade the standby never heard of.

But here's the catch. The standby takes over from 109. That trade never happened. But the member saw it. Market data too. Maybe clearing.

The fix: every output waits at a gate. Until its input is safely on the hot standby. And on the near site too. It copies in real time, aiming for near-zero data loss. Only then: the ack, the trade, the tick.

The standby replays the same log, with the same code. Both hash their books and compare. A mismatch? Not fit to take over. And snapshots every few minutes keep a restart short.

Latency can rise for a while. Lost data can't be undone. Never acknowledge what a surviving copy can't replay.

Full breakdown in the lecture.