Rogers deleted one routing filter and Canada lost a whole carrier, 911 included.

On 8 July 2022, Rogers, one of Canada's biggest carriers, went dark: mobile, internet, business lines and, for its customers, 911.

For weeks Rogers had been upgrading its IP core network in a seven-phase process. That morning was phase six. In its response to the regulator, the CRTC, as quoted by The Register, Rogers wrote: "At 4:43AM EDT, a specific coding was introduced in our distribution routers which triggered the failure of the Rogers IP core network starting at 4:45AM." According to that response, the change deleted a routing filter and let all possible routes to the Internet through. The routers began pushing "abnormally high volumes of routes" into the core, the core equipment exceeded its capacity and stopped routing, and wireless and wireline lost the Internet together, because they shared one core.

A review for the CRTC by Xona Partners, reported by The Canadian Press, found why the change got through. The first five phases had gone fine, so a risk algorithm rated phase six low risk, which let it skip extra scrutiny and approvals. The review said overload protection limiting how much routing information flows into the core could have prevented the failure.

Recovery was slow. According to the review, with the core down, remote staff couldn't reach Rogers' systems, so people went to sites to reach routers in person. Critical error logs showing the root cause couldn't be accessed until 14 hours in. Rogers said service came back from the afternoon into the evening.

The next day, CEO Tony Staffieri pointed to "a maintenance update in our core network". Two weeks later he promised C$10 billion over three years on reliability, and to separate wireless and internet.

Why it matters: Past success doesn't make the next change low risk. Cap what one router can push into the core, and keep a way into your systems that doesn't run through them.