One server missed a deploy. Knight lost $460 million in 45 minutes.

transcript

About 45 minutes. Over 460 million dollars. Gone. Because one server out of eight missed a deploy.

Knight did about ten percent of all trading in listed US stocks. Its system held old code: Power Peg. In 2005, the part that told it to stop moved elsewhere. Nobody retested. 2012. Knight writes new code to replace Power Peg. And the new code reuses Power Peg's old on-switch. Its flag.

From 27 July, the new code goes out in stages. A technician misses one of eight servers. Nobody checks. 1 August, 8:01 a.m. 97 emails mention Power Peg disabled. Nobody acts. Flagged orders reach server eight. Power Peg wakes up.

On server eight, the flag still meant Power Peg. It sends child orders. Again and again. Never counting the fills. Nothing tells it to stop.

Staff hunt for the cause. They pull the new code from the seven good servers. Now Power Peg runs there too. And no plan says when to unplug.

About 45 minutes. 212 customer orders go in. Over 4 million executions come out. Knight lost over 460 million dollars.

Dead code on a production server is still live code. Delete it. Never reuse an old switch. Check every machine got the deploy.

Full breakdown in the lecture.