Summary

  • At 07:08 UTC on 25 January 2023, an unqualified command used during capacity work in Madrid purged routing information beyond the local device on one vendor's platform, triggering global IGP recomputation, BGP re-advertisement and the first wave of customer impact.
  • The operation was repeated on a second Madrid router 33 minutes later because the engineer had not been informed of active alerts, producing a second wave and turning change-to-response coordination into a central accountability issue.
  • Most regions and services recovered by 09:05 UTC and the final networking equipment by 09:25, but paused WAN-health and traffic-engineering systems required manual restarts before full mitigation at 12:43; recovery was therefore staged rather than a single rollback event.

A bounded task reached a global routing domain

At 07:08 UTC on 25 January 2023, an engineer in Madrid began work to add capacity to Microsoft's global wide-area network. The task involved changing IP addresses on new routers and bringing them into the network's internal and external routing domains. The physical work was limited to a pair of routers. The logical reach of the command used during that work was not.

Microsoft's final post-incident review says the procedure contained an instruction that purged the Interior Gateway Protocol database. On equipment from two manufacturers, that instruction operated locally. On a third manufacturer's platform, its default scope extended to every router connected through the IGP. The company's command-authorization system had not classified that platform-specific behaviour as something to block.

The mismatch set off a network-wide recomputation of internal topology. Border Gateway Protocol routers then re-advertised and validated Internet prefixes as paths changed. Customers experienced a shifting mix of latency, timeouts, intermittent packet loss and, on some paths, complete loss of connectivity. The incident affected Internet-to-Azure traffic, inter-region traffic and cross-premises links using ExpressRoute and VPN or virtual WAN. Services dependent on Azure connectivity, including parts of Microsoft 365, Power Platform and Azure Government, were also affected.

This chain makes the incident a network-infrastructure accountability case rather than a generic cloud interruption. Remove the IGP purge, the BGP re-advertisements and the changing forwarding paths, and the central explanation disappears. The system-level question is not simply why a person entered a command. It is why the surrounding procedure and authorization controls could not establish the command's effective scope before it reached the live routing domain.

The failure also shows why an apparently routine capacity change deserves the same semantic discipline as a more visibly disruptive operation. A task label describes intent. A router executes software. When the two disagree, the running implementation determines the blast radius, regardless of how narrow the work order appears.

The procedure described one task; the routers executed another

According to Microsoft's final review, the standard operating procedure used for the Madrid capacity work had been changed before the event but had not been re-tested. It also lacked required checks before and after execution. Those omissions mattered because the same words did not carry the same operational scope on every platform.

The distinction between a local and IGP-wide command is not a documentation footnote. In a global WAN, an instruction that reaches all IGP-connected routers changes the set of devices calculating internal reachability at the same time. The initiating failure described by Microsoft was inside the WAN: a broadly effective IGP purge triggered topology recomputation, followed by BGP re-advertisement and route validation.

The command was allowed because Microsoft's authorization controls did not identify the third platform's global behaviour as prohibited. This separates permission from qualification. A command can be accepted by an access-control system and still be unsafe for the context in which it is used. Authorization answers whether an operator may act. Qualification must answer what the network will do when that action reaches a particular platform, software environment and device role.

A text-level allow list cannot supply that answer on its own. Two vendors may use similar language for commands whose default scope is materially different. Even within one vendor family, a safe conclusion depends on the software and the role the device occupies. The incident record does not provide a full inventory of those variables, so it would be wrong to infer which exact platform or version was involved. It does establish that the control system missed a manufacturer-specific global effect.

That is where organisational accountability begins. A multi-vendor command library needs more than approved strings. It needs evidence that scope has been tested for the environments in which the command may run, that the intended boundary can be checked before execution, and that the observed network state can be compared with that expectation after execution. Without those elements, approval can become a statement about identity and access rather than a safety case for the change.

The missing pre- and post-checks amplified the semantic gap. A pre-check could have required the change owner to state the expected device and routing scope. A post-check could have compared actual topology and reachability with that declaration. These are control-design implications drawn from the event, not claims that any particular check would have prevented every failure. Their value is that they make scope falsifiable before a second action repeats the same assumption.

Public routing data showed the disturbance, not the root cause

External observers saw the network react. ThousandEyes recorded Microsoft prefixes being withdrawn and re-advertised, traffic moving away from direct peering links toward transit providers, and substantial packet loss. Kentik reported two aggregate traffic drops and highly variable effects between paths and regions. Some prefixes were impaired while others continued to carry traffic.

Those observations matter because they show the failure at the boundary customers and neighbouring networks could measure. They also impose a limit on the conclusions that public BGP data can support. Kentik cautioned that the visible routing instability was likely a symptom of the internal event, not its origin. Withdrawals and transit shifts do not demonstrate a malicious route hijack, an external attack or a BGP design flaw.

BGP was nevertheless central to the customer-facing failure chain. Once the internal topology was being recomputed, border routers had to re-advertise and validate Internet prefixes. Paths changed, and some traffic shifted away from direct peering toward transit. To a customer, that control-plane movement appeared as latency, loss or an unreachable service. To an outside observer, it appeared as withdrawals, announcements and path changes. Neither view, by itself, revealed the internal command.

The distinction is important for post-incident accountability. External telemetry can verify that reachability changed and can help reconstruct which paths disappeared, returned or moved to transit. It cannot, by itself, identify the internal instruction, the decision path that allowed it, or the state of proprietary control systems. Microsoft's final review supplies that internal mechanism. The outside measurements test whether the described mechanism is consistent with what the Internet saw.

The records therefore perform different evidentiary jobs. The operator's account explains the internal sequence and the status of its own systems. ThousandEyes and Kentik provide independent observations of route and traffic behaviour. Agreement between those layers strengthens the causal account, while disagreement would be a reason to investigate further. Treating one layer as a substitute for the other would weaken the analysis.

The same discipline applies to preliminary explanations. Early outside analysis considered whether automation might have repeated the change. That was a reasonable question while the incident record was incomplete, but it is not the controlling conclusion. Microsoft's final review states that the operation was performed again on the second router. Later, more specific evidence must take precedence over an earlier hypothesis.

Thirty-three minutes turned one control failure into two waves

The event did not unfold as one uninterrupted automatic cascade. Microsoft says monitoring detected DNS and WAN symptoms within minutes, generating alerts at 07:11. Yet the engineer carrying out the Madrid work was not informed of those alerts and repeated the operation on the second router 33 minutes after the first. That second operation produced another impact wave.

Both ThousandEyes and Kentik observed distinct routing or traffic disturbances, supporting the two-wave account. Their measurements cannot identify the operator or the internal communication path, but they corroborate the separation visible outside the network. The final review supplies the reason the second operation occurred under the same procedure.

The 33-minute interval matters because it reveals a control boundary between change execution and incident response. Monitoring detected a problem. Response processes began. The person able to invoke the same mechanism again did not receive the information needed to stop. Each component may have been active, yet the combined system did not prevent a second execution while the network was already recovering.

An alert that reaches a monitoring platform but not the person continuing a risky operation is an incomplete safeguard. For high-impact WAN work, detection has limited protective value unless it can change the state of the work: pause the procedure, revoke the next step, require a fresh authorization, or place an incident owner in control. The public record does not specify which of those controls existed or failed beyond the notification gap, so the choice among them remains a design question.

This is also why individual blame is a poor substitute for analysis. The public record does not name the engineer, disclose the operator's full console context or establish personal intent. It does establish a procedure that had not been re-tested, missing pre- and post-checks, incomplete command classification and a notification path that did not stop the second operation. Those are system properties that an infrastructure owner can inspect and change.

The second wave added more than another period of disruption. It complicated recovery by introducing a new disturbance while convergence from the first was in progress. The sources do not quantify the incremental harm attributable to each wave, so no such number should be inferred. What can be said is narrower: the same unqualified operation was able to reach the routing domain twice, and outside observers saw two distinct effects.

Impact varied with the path

The incident was global in scope, but its customer effects were not uniform. Microsoft describes dynamic differences by source and destination. Some customers saw latency or intermittent packet loss; others encountered timeouts or complete loss of connectivity on particular paths. Kentik likewise found affected and unaffected prefixes rather than evidence that every Microsoft route disappeared.

That variability is consistent with a network in which routes are being withdrawn, re-advertised, validated and shifted across different paths. It does not support a simple claim that every Azure region, service, customer or Internet route failed. India and parts of North America were among the longer recovery paths, but the sources do not provide a complete region-by-region inventory.

The supported impact surfaces are substantial without exaggeration. Internet-to-Azure traffic and inter-region traffic were affected. Cross-premises connectivity through ExpressRoute, VPN and virtual WAN experienced problems. Microsoft 365 and Power Platform dependencies, along with Azure Government services dependent on public Azure, were also involved. The record does not establish an exact affected-user count, an audited revenue loss or a complete service list.

Path variability is part of the accountability problem because aggregate availability can hide pockets of continuing harm. A broad recovery percentage may look healthy while specific source-destination pairs remain impaired. The incident's staged recovery shows why operators need both network-wide indicators and path-level evidence before declaring the customer experience restored.

Recovery came in layers

Microsoft says engineers reviewing recent changes identified the problematic command by 08:20, while routing recovery was already progressing. Nearly all network devices, regions and services had recovered by 09:05, and the last networking equipment recovered by 09:25. That was not the end of the incident.

Some locations, including India and parts of North America, followed longer recovery paths. Localised packet loss persisted because WAN-health and traffic-engineering systems had paused. Those systems required manual restarts before Microsoft declared full mitigation at 12:43. The official customer-impact window therefore runs from 07:08 to 12:43, even though most services recovered much earlier.

The Register's contemporaneous report captured the visible Azure and Microsoft service disruption while the event was still developing, along with Microsoft's early references to routing changes and rollback. Those initial statements are useful evidence of what customers were being told at the time. They should not replace the later incident review's more complete clock or its account of automatic convergence and manual restarts.

Describing the recovery as a single rollback would erase the operational work that followed the first restoration of routes. A network can regain broad reachability while health measurement and traffic engineering remain degraded. Recovery evidence therefore needs to cover not only whether packets can move again, but whether the systems that detect unhealthy paths and steer traffic have returned to a known operating state.

The distinction also protects incident timelines from becoming misleadingly short. A 09:05 milestone describes the point by which nearly all devices, regions and services had recovered. A 09:25 milestone describes the final networking equipment. The 12:43 boundary accounts for the supporting systems and lingering packet loss. Each clock answers a different question, and none should be silently substituted for another.

A staged timeline supports a more rigorous definition of recovery. First, routing convergence must restore broad reachability. Second, the remaining equipment and paths must return. Third, health and traffic-engineering functions must be restarted and verified. The sources establish that sequence for this incident, but they do not prove how Microsoft now defines recovery or whether its later repair commitments were completed.

The record supports a system finding, not a legal verdict

The four-source record supports a specific chain: a changed, untested procedure included a command whose scope differed across router manufacturers; the command triggered broad IGP recomputation; BGP advertisements and paths changed; customers saw uneven connectivity failures; and the operation was repeated before the engineer was informed of the alerts, creating a second wave. Recovery then proceeded through routing convergence and later manual restart of supporting systems.

Microsoft's review listed planned work that included auditing high-impact commands across vendors and roles, expanding the block list, making approved and unqualified device activity visible to responders, requiring operational training and attestation, and reviewing pending procedures through a change-advisory process. The public record used here does not independently establish that each measure was completed or that it has since proved effective. They remain stated repair commitments, not verified outcomes.

The record does not establish how many customers or users were affected, a complete inventory of every impaired region or service, an audited financial loss, customer data loss, malicious activity, or a legal finding of negligence or liability. It also does not disclose enough internal logs to allocate every decision to a named owner or to assess every compensating control that may have operated during the incident.

Those limits do not weaken the infrastructure finding. They keep it properly scoped. Microsoft's 2023 WAN outage showed that a global routing system can be disrupted when procedure, authorization and platform semantics disagree about the reach of a command. The measure of control is not what the change was called or who was permitted to type it. The measure is whether the live network's actual behaviour was bounded, observable and recoverable before a second wave became possible.

Sources