One expired certificate in Ericsson software cut mobile data in 11 countries at once.

Mobile operators buy the machines at the heart of their networks from equipment makers like Ericsson. One of them is the SGSN-MME, a node in the core network, the central part of a mobile network. Operators in many countries run the same Ericsson software on it.

On Thursday 6 December 2018, mobile data failed on several of those networks at once.

In the UK, O2's data network went down, along with the brands that run on it, including giffgaff, Sky Mobile, Lyca and Tesco Mobile: about 32 million people, The Register reported. A Transport for London timetable service was among the knock-on casualties.

In Japan, SoftBank's 4G was out nationwide from 1:39 p.m. to 6:04 p.m. local time, across the SoftBank and Y!mobile brands, its home phone service and part of SoftBank Air. Traffic surged into the 3G network and made that hard to use too. SoftBank's network centre found the software failing on all its Ericsson packet switches, in both its Tokyo and Osaka centres.

Ericsson's statement that day: "An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers." Two specific software versions were involved. SoftBank added that Ericsson told it the same software had been running for nine months, and that the same failure hit carriers in 11 countries at once.

SoftBank recovered by putting the older version of the software back on every packet switch. Ericsson's CEO, Börje Ekholm, said the faulty software "is being decommissioned and we apologize not only to our customers but also to their customers."

Why it matters: A certificate baked into shipped software is a timer nobody sees. When every customer runs the same version, it goes off everywhere at once. Track the expiry dates inside what you ship, not only on your websites.