To keep the atomic clocks in step with the Earth's rotation, timekeepers sometimes add a leap second. One went in at midnight GMT as 30 June 2012 turned into 1 July.
At Reddit, system administrator Jason Harvey first blamed the network: a massive Amazon cloud outage had hit less than 24 hours earlier. After about half an hour, his team traced it to their own Linux servers, which had almost ground to a halt. Their Cassandra database, built with Java, had processes stuck in constantly spinning loops, eating the CPU. Reddit rebooted its servers. The site was all but inoperable for about 30 to 40 minutes and fully offline for about an hour and a half. Gawker had trouble with its Tomcat web servers, and Mozilla with Hadoop.
The cause was in the Linux kernel. When the kernel added the leap second, it didn't tell its high-resolution timer system, hrtimer, that the clock had changed. Programs that sleep with a short timeout rely on those timers. As kernel developer John Stultz explained, timers fired one second early, and timeouts under a second fired immediately. Programs that waited, woke and waited again now looped flat out: load spikes, often with MySQL, Firefox, Thunderbird and Java.
The quick workaround was a one-line command that set the clock to the current time, which forced the missing notice. On 5 July 2012, Stultz submitted his fix for inclusion: the kernel would now send that notice whenever it adds or removes a leap second.
Linus Torvalds told WIRED: "Almost every time we have a leap second, we find something." It is, he said, "code that is basically never run".
- 1 extra second, at midnight GMT on 30 June 2012
- About half an hour for Reddit to trace the problem
- Reddit all but inoperable for 30 to 40 minutes, fully offline for about an hour and a half
- Timers fired 1 second early; sub-second timeouts fired immediately
Why it matters: Code that only runs once every few years is barely tested. Rehearse rare events like leap seconds and date changes on purpose, before the real one arrives.