Rogers deleted one routing filter and lost its whole network, 911 included

transcript

Rogers deleted one routing filter. Two minutes later, its core began to fail. Mobile. Internet. Business lines. Even 911.

Rogers is upgrading its core network in seven phases. Five go fine. So phase six is rated low risk. Fewer checks. Fewer approvals. 4:43 a.m. The change reaches the distribution routers. 4:45 a.m. The core network starts to fail.

The villain? The change deleted a routing filter. Now every possible route to the Internet flows through. The core gets flooded. It runs past capacity and stops routing. Mobile and internet share that one core. Both go down together.

But here's the catch. The engineers needed that network too. With the core down, remote staff couldn't reach the systems. People had to go to sites and reach routers in person. The logs showing the root cause? Out of reach for 14 hours.

About 12 million people lost service. Rogers then promised 10 billion Canadian dollars over three years.

Five clean phases don't make the sixth one safe. Cap what one router can push into the core. And keep a way in that doesn't run through your own network.

Full story in the article.