One extra second jammed Linux servers across the web

transcript

One extra second jammed Linux servers across the web. Reddit, Gawker and Mozilla. All hit trouble at midnight.

To match Earth's rotation, clocks sometimes get a leap second. Reddit's admin blames the network. A massive Amazon outage hit less than a day earlier. Half an hour later, the truth. Their own Linux servers, nearly frozen. Cassandra processes are stuck in spinning loops, eating the CPU. They reboot.

About half an hour just to find the problem. The site was all but unusable for 30 to 40 minutes. Fully offline for about an hour and a half. Over one second.

The bug was in the Linux kernel. It added the second to its clock. But it never told its timer system, hrtimer. So timers fired one second early. Sub-second timeouts fired immediately. Programs that wait, wake, and wait again? Now they loop flat out. Load spikes, often with MySQL, Java, even Firefox.

The quick fix was one command. Set the clock to the current time. That forced the missing notice through. 5 July: the real fix. The kernel now sends that notice every leap second.

Linus Torvalds said it to WIRED. Almost every time we have a leap second, we find something. Why? It's code that is basically never run.

Code that runs once every few years is barely tested. So rehearse leap seconds and date changes on purpose. Before the real one.

Full breakdown in the lecture.