Summary

  • The CRTC assessment places the outage from 04:58 EDT on July 8 to 07:00 EDT on July 9 and reports that more than 12 million wireless and wireline customers lost service [1].
  • During phase six of a seven-phase core upgrade, staff removed an ACL policy filter; the report says full BGP tables were then redistributed into OSPF and exhausted CPU and memory on core routers [1].
  • Earlier upgrade phases had succeeded, and the triggering phase was downgraded from High to Low risk, removing requirements for added scrutiny, higher approval, and laboratory testing [1].
  • Rogers' management network depended on the failed IP core, while critical sites lacked alternate-provider management connectivity; that delayed diagnosis and physical access became necessary [1].
  • A credible closeout must prove route-volume bounds, configuration-diff review, automated rollback, management independence, emergency-service outcomes, and a controlled negative recurrence test.

What happened

The later CRTC assessment provides the strongest public technical account. It says Rogers operated wireless and wireline access networks through a common IP core. In the weeks before July 8, the company was executing a seven-phase upgrade of that core. The outage occurred during phase six [1].

The triggering change removed an Access Control List policy filter from distribution-router configuration. According to the assessment, this enabled full Internet BGP routing tables to be redistributed into OSPF. BGP carries reachability between autonomous systems; OSPF distributes internal topology and reachability inside an operator. Importing an uncontrolled external route set into an interior protocol can turn a valid-looking configuration action into an enormous state increase [1].

The report says the resulting updates exhausted CPU and memory resources in the core routers. The policy filter was removed at 04:43 EDT. Within two minutes, core gateways began failing. By 04:58, the report's outage timeline records distribution routers flooding the core with more routes than it could process [1]. Mobile, home phone, Internet, business wireline, and 9-1-1 connectivity ceased functioning. Restoration proceeded gradually until 07:00 the next morning [1].

Rogers' early public messages were necessarily less specific. On July 8, the chief executive acknowledged failure across wireless and wireline service and accepted responsibility [4]. On July 9, Rogers said it had narrowed the cause to a core-network maintenance update that made routers malfunction, and that teams had disconnected specific equipment and redirected traffic while service returned [5]. Those statements are useful contemporaneous evidence. The later CRTC assessment supplies the detailed route-redistribution mechanism and should govern technical claims.

The failed boundary was between routing systems

This was not merely a case of one router receiving too much customer traffic. The reported failure crossed a routing-protocol boundary. BGP and OSPF serve different scopes. BGP can carry very large Internet route sets and policy-rich paths. OSPF is designed to let routers inside one administrative domain build a consistent link-state view. Moving selected routes between them can be legitimate. Moving an unbounded full table into the interior protocol can force many devices to process state that was never intended for that domain.

The accountability surface begins with an explicit redistribution contract. The operator should be able to name which routes may cross from BGP into OSPF, why they are needed, their maximum count, which attributes are preserved, which devices receive them, and what happens when the count exceeds expectation. A policy filter is not just configuration syntax. It is the executable boundary between two state systems.

The CRTC assessment says Rogers' standard distribution-router configuration permitted BGP routes to enter OSPF when the policy statement no longer blocked them. It identifies four missing or inadequate controls relevant to the event: overload protection on core routers, limits on BGP routes advertised into OSPF, manual and automated policy-command audits, and automated rollback [1]. The report also says the core routers had no configured overload limit for the routing information they could accept [1].

That matters because a redundant pair can reproduce the same failure. If every core device receives the same unbounded route flood, redundancy duplicates exposure instead of creating independence. A topology can have multiple routers, links, vendors, and sites while retaining one shared policy failure domain. The correct test is not how many devices exist. It is whether one invalid state can cross the boundary and exhaust all of them before a limiter, validator, or rollback stops it.

Change risk followed the successful phases downward

The configuration error was also a governance failure in executable form. The CRTC assessment says Rogers initially rated the seven-phase process High risk. After earlier phases succeeded, the risk algorithm downgraded phase six to Low, including the change that removed the route filter. That lower rating meant the change did not require additional scrutiny, higher approval levels, or laboratory testing [1].

Success in an earlier phase is not proof that a later phase is equivalent. A staged program may touch different devices, policy branches, traffic volumes, or failure domains at each step. A change that removes a filter is especially asymmetric: the configuration diff can be small while the reachable state space becomes much larger. Risk scoring must inspect the semantics of the current diff, not reward the program for having survived previous work.

A strong change record would therefore bind the ticket to the exact pre-change and proposed configuration, a machine-readable route-policy intent, lab output, expected route counts, approval, deployment scope, observation window, rollback trigger, and owner. The reviewer should see that a filter is being removed and ask what previously depended on it. Automated validation should reject a route-count increase beyond a declared envelope before production rollout.

Canarying also has to match the failure mode. Applying a change to one device is useful only if that device can be isolated and its route state observed before peers receive the same update. A canary that immediately exports the unsafe state to the common core is not a small blast radius. The rollout plan should prove that the test device cannot contaminate every redundant peer.

A common core made the consequence national

The assessment describes a common IP core carrying both wireless and wireline services. It explicitly says the outage was not caused by a design flaw in an otherwise typical Tier 1 architecture. It also says convergence made the scope extreme because the common failure removed both access families at once [1]. These two findings should remain together.

Convergence can reduce duplication and improve utilization. It can also concentrate voice, mobile data, home Internet, business connectivity, emergency calls, public alerts, monitoring, and internal communications behind a shared routing decision. The issue is not whether convergence is categorically wrong. The issue is which services must survive one control-plane failure and which dependencies need separate failure domains.

The CRTC executive summary says Rogers later decided to separate wireless and wireline IP cores, leaving the existing core for wireline and deploying a new core for wireless. At the time of that report, implementation was not yet complete [2]. The assessment presents separation as a way to contain a future failure to one access network. That is an architectural control, but it still requires independent policy, change rollout, management access, and testing. Two cores administered by the same unsafe automation can remain correlated.

Emergency communications sharpen the standard. The detailed report says 9-1-1 connectivity and public-alert delivery were affected. Rogers notified 9-1-1 network providers at 08:39, three hours and 56 minutes after outage onset, and sent its first customer message at 08:54 [1]. The CRTC's July 12 letter criticized the lack of early guidance on alternate access to 9-1-1 and demanded a comprehensive account [3]. Exact call-success figures are redacted in parts of the report, so a public article should not invent them.

The management network failed with the data network

The most consequential recovery problem was loss of control access. The CRTC assessment says Rogers' management network relied on the same IP core that failed. Remote employees could not access critical network elements, and the network operations centre and other important sites lacked redundant management connectivity from alternative providers. Staff had to travel to equipment for physical access, delaying diagnosis and repair [1].

The operator's own communications also depended heavily on its services. The report says incident personnel had limited third-party mobile SIMs and struggled to communicate when both fixed and mobile networks failed. Error logs from failed routers were not initially available, and the root cause was not pinpointed for about 14 hours [1]. Several changes had been made in the maintenance window, making it harder to identify which ticket should be reversed.

This is a strict independence test. A management plane that rides the production data plane is convenient during normal operation, but it is not a recovery path for a common-core failure. A credible design needs separate physical and logical access, independent credentials, tested alternate-provider links, locally reachable console paths, and incident communications that do not depend on the affected operator.

RFC 6192 discusses router control-plane protection as a general engineering principle: identify legitimate control traffic, filter or rate-limit other traffic, and preserve control-plane stability despite load [8]. It does not diagnose Rogers' event or specify BGP-to-OSPF redistribution policy. It does reinforce a broader point: forwarding capacity and control capacity are different resources, and the control system must remain stable when data or routing load increases.

Recovery evidence must connect commands to service

The public assessment records meaningful changes after the incident. Rogers added safeguards to prevent route-data flooding, built a physically and logically separate management network, added third-party backup connectivity at critical sites, invested in configuration-validation tools, revised its risk algorithm, increased lab testing, improved rollback automation, clarified incident roles, prioritized alarms, and supplied incident teams with backup communications [1][2]. The assessment judged the combined measures satisfactory for addressing the root cause and improving resilience [2].

Those are control claims. An accountable closeout also needs running evidence. For route safety, the evidence is the configured redistribution ceiling, the observed BGP and OSPF route counts, a rejected over-limit test, and proof that core CPU and memory remain bounded. For change safety, it is the exact diff, independent approval, lab result, canary observation, and automated rollback under a failed assertion.

For management independence, the evidence is an exercise in which the production core is unavailable while engineers still authenticate, reach devices, retrieve logs, coordinate, and execute a reversible change. A diagram or carrier contract is not enough. The alternate path must work under the same power, identity, and staffing constraints that apply during an incident.

For service continuity, the evidence is external. Test wireless registration, voice, SMS, data, fixed Internet, business access, 9-1-1 routing, public alerts, and representative payments or cloud transactions. Record each service separately. A restored route does not prove a call completes, and a green router does not prove an emergency alert reaches a handset.

The registry records identity, not continuity

Current ARIN RDAP records identify AS812 as ROGERS-COMMUNICATIONS and list Rogers Communications Canada Inc. as registrant [6]. Current PeeringDB data identifies AS812 as Rogers Cable and publishes present-day interconnection attributes [7]. These records establish a public network identity and a contact surface. They do not reconstruct the private July 2022 core, prove historical peers, or show that a route filter remained in place.

That distinction is the Heng.lu doctrine surface. A registry is a ledger and recordkeeper. It can say who holds an autonomous system number and what an operator currently publishes about its network. Running code and observed route state decide whether traffic moves. The accountability record has to connect the ledger identity to the configuration, route count, device state, service result, and responsible owner at a specific time.

The same rule applies to change systems. A ticket marked approved is not proof that the deployed diff matched the review. A rollback command marked successful is not proof that unsafe routes withdrew. A redundant-path inventory is not proof that management access survived. Each administrative record needs a corresponding observation from the running network.

What a credible recurrence test would contain

First, stage the exact policy class in a lab with production-scale route volume. Remove or alter the redistribution filter under controlled conditions. Confirm that a hard route ceiling rejects the state before core resources rise materially. Verify that alarms identify the device, protocol boundary, route count, and triggering change.

Second, run a production canary whose exports cannot reach the whole core until assertions pass. Compare intended and observed BGP-to-OSPF routes. Force one assertion to fail and prove automatic rollback restores the previous configuration without operator improvisation.

Third, isolate the production IP core in an exercise. Require responders to use the separate management path, alternate-provider communications, independent identity service, and locally retained logs. Measure time to access, time to diagnosis, time to a safe change, and time to external service confirmation.

Fourth, test common-mode service failure. Remove the shared component or policy path and prove the intended wireless/wireline separation. Confirm 9-1-1 and public-alert behavior with the responsible external organizations, not only internal simulators. Record cases where a handset remains attached to a radio network but cannot complete an emergency call.

Finally, reconcile every layer into one timeline: approved ticket, deployed bytes, route changes, device resource use, alarms, responder access, rollback, external probes, emergency-service outcomes, and customer communications. The test passes only when these records agree or explain their disagreement.

What to watch next

The first signal is evidence that route redistribution has a declared and enforced maximum. Operators should be able to detect an unexpected increase before OSPF state reaches every core router. The second is whether change-risk scoring treats filter removal, protocol redistribution, and common-core reach as inherently high scrutiny even after earlier phases succeed.

The third signal is tested management independence. Backup links and third-party SIMs should appear in exercises with expiry, ownership, capacity, and authentication checks. The fourth is service-specific recovery reporting: wireless, wireline, business, emergency calls, public alerts, and internal operations should not be compressed into one restored timestamp.

The lasting lesson is narrower than saying a bad configuration took down Canada. Public evidence supports a specific chain: a route-policy filter was removed; full BGP tables entered OSPF; common-core routers were overloaded; shared services and management access failed; diagnosis and restoration were prolonged. Accountability means proving that the next equivalent state is bounded before it can become a national outage.

Sources

  1. https://crtc.gc.ca/eng/publications/reports/xonarp2023.htm
  2. https://crtc.gc.ca/eng/publications/reports/xona2024.htm
  3. https://crtc.gc.ca/eng/archive/2022/lt220712.htm
  4. https://about.rogers.com/news-ideas/a-message-from-tony-staffieri-president-and-ceo-at-rogers/
  5. https://about.rogers.com/news-ideas/a-message-from-rogers-president-and-ceo/
  6. https://rdap.arin.net/registry/autnum/812
  7. https://www.peeringdb.com/api/net?asn=812
  8. https://www.rfc-editor.org/rfc/rfc6192.html