OVH's 2017 Optical-Network Failure Made Redundancy Independence an Acc

Summary

The event boundary is narrow: This article covers only the Roubaix optical-network failure of 9 November 2017. The simultaneous Strasbourg electrical outage was a separate incident with a different mechanism and control chain.
The visible failure was broad but not universal: Contemporary accounts described lost Roubaix connectivity toward six of OVH's network points of presence. That does not prove every OVH route, customer or workload failed in the same way.
Nominal diversity did not provide operational independence: OVH described redundant optical connectivity, yet the Roubaix links were unavailable together after configuration loss and had to be restored from saved configuration.
The public root cause has two layers: OVH attributed the immediate event to a software defect and loss of optical-equipment configuration. The correlated loss also exposed a shared configuration or supervisory failure domain that physical path diversity had not removed.
Saved configuration is not an independent recovery system: A saved configuration enabled restoration, but its existence did not keep the running optical system available. Backup availability, restore authority and tested recoverability must be measured separately.
Responsibility follows control: OVH controlled architecture, deployment, configuration protection, monitoring, restoration and customer communication. The equipment maker controlled product-defect investigation and software correction. Peers, transit networks and customers controlled their own external diversity, but not OVH's internal optical state.
The announced repair correctly separated two problems: OVH planned a software upgrade with the equipment maker and a split of the optical multiplexing system across two systems. The first addressed a defect; the second was meant to limit recurrence blast radius.
A repair is not proven by an announcement: Credible closeout requires topology and configuration evidence, failure-injection tests, independent reachability measurements and proof that one control failure can no longer remove all Roubaix external paths.

Freeze the Roubaix event before assigning responsibility

OVH experienced two serious failures on the same day. A power failure affected its Strasbourg site, while an optical-network failure affected Roubaix. OVH's later statement explicitly described the incidents as simultaneous but unrelated. That distinction is the starting point for accountability, not a detail to be collapsed into a larger story. [1]

Combining the incidents would produce an inaccurate causal chain. Strasbourg involved electrical supply, transfer equipment, generators and service restart. Roubaix involved optical transport, configuration loss and connectivity toward external network locations. The responsible systems, operators, vendors, detection signals, recovery actions and prevention tests were different.

This article therefore starts at the Roubaix optical failure and ends when the relevant links and hosting-site connectivity were restored, followed by the specific Roubaix remediation OVH announced. It excludes the Strasbourg power event, a July 2017 storage incident, a later December 2017 network incident, the 2021 Strasbourg fire and a separate 2021 routing event. Those events may inform a broader history of OVH, but they cannot be used as evidence for the cause or repair of the Roubaix optical failure.

The public chronology has two useful levels. OVH's formal statement said the Roubaix site returned to operation in under two and a half hours. An affected service provider separately recorded customer-visible unreachability during the same morning and described restoration after the optical configuration was recovered. [1][4]

Those accounts describe the optical recovery window at different levels. They do not establish that every customer service recovered at one exact moment. A hosting site is a stack: transport links return, routing sessions re-establish, paths converge, load balancers and application dependencies recover, mail queues drain, monitoring clears and customers retry. Public evidence does not supply a complete service-by-service restoration table.

The same discipline applies to the start of the incident. An affected service provider described accessibility from some networks but not others during the morning disruption, while OVH described the provider-side optical failure and later restoration. These perspectives are not necessarily contradictory. One records an external customer's observed effect; the other records an event in the provider's optical system. Different observers, clocks and network paths can see different boundaries. [4]

Accountability requires preserving those distinctions instead of forcing one perfect time. A strong incident record would align external probes, optical alarms, interface transitions, controller logs, routing-session changes and customer reports against a common clock. The public material does not provide that complete alignment, so this article does not invent it.

What the running network exposed

The Roubaix site depended on optical transport connecting it toward six of OVH's network points of presence, according to the affected service provider's account. The public record describes multiple optical paths and broad loss of external connectivity, but it does not provide a complete verified topology for every circuit, customer path or dependency. [4]

Those details matter because they show that the event was not presented as a routine single-fibre cut. OVH and contemporary accounts instead described a software and configuration failure affecting the optical system, followed by recovery work with the equipment manufacturer. The available sources do not expose the complete diagnostic sequence or internal management-plane state. [1][4]

The public root-cause description identified configuration loss in the optical equipment. OVH recovered saved configuration and restored the affected connectivity. The company attributed the immediate event to a software defect, while the correlated path loss exposed a wider question about shared configuration and supervisory dependencies. [1][4]

That is a meaningful account, but it is not a complete forensic report. It does not disclose the exact software version, initiating state change, process crash, storage sequence, replication semantics, control-plane election, human command, alarm chronology or internal incident record. It does not prove whether the lost configuration was the sole initiating cause or one visible consequence of a deeper control failure.

The strongest defensible statement is narrower: OVH's optical equipment entered a state in which the Roubaix external links were unavailable; OVH attributed that state to a software defect and missing configuration; restoration of saved configuration returned connectivity; and OVH later proposed both software remediation and architectural separation.

The running network is the primary evidence. Inventory records can say that fibres, paths, cards and backups exist. Design documents can say that links are redundant. What mattered during the incident was that the optical system stopped providing the external connectivity required for Roubaix reachability. The operational state overruled the nominal diagram.

This is the central network-infrastructure accountability issue. Redundancy should not be counted by components alone. It should be evaluated by the failure domains that can remove service.

Physical diversity is not control-domain independence

The common visual model of resilient transport is two lines between two locations. If one fibre is cut, traffic uses the other. That model is useful for a narrow physical hazard. It is incomplete when both paths share software, configuration authority, supervisory hardware, timing, power, management access, activation logic or a common recovery procedure.

Two geographically distinct optical paths can still occupy one operational failure domain. They may terminate on cards governed by the same database. They may depend on the same controller or supervisory pair. They may receive the same faulty software image. They may inherit one configuration transaction. They may require one management network for diagnosis. They may fail closed when a common safety state is entered.

OVH's account is a concrete example of that distinction. The provider described redundant optical connectivity, yet the Roubaix links were unavailable together when configuration was lost. The controls intended to preserve connectivity did not contain the shared configuration-state failure that actually occurred. [1][4]

This does not mean the physical diversity was useless. It means it addressed a different hazard. Resilience claims should name the hazard class they contain:

a single fibre cut;
a conduit or geographic route failure;
loss of one optical amplifier or node;
loss of one line card or chassis;
failure of one controller or monitoring card;
corrupt or missing configuration;
a common software defect;
loss of management connectivity;
operator error propagated across redundant systems;
restoration failure under incident conditions.

A design may pass the first three and fail the sixth or seventh. Calling the result "redundant" without naming the protected failure class conceals the most important question.

RFC 3439 warns, in general architectural terms, that complexity has real costs and that systems often fail in unexpected interactions. It was not written about OVH's incident and does not prove the provider's internal design. It offers a useful analytical discipline: adding components, copies and automation can create shared state and additional failure modes unless their behavior is bounded and observable. [14]

NIST's cyber-resilient-systems guidance similarly treats resilience as an engineered ability to anticipate, withstand, recover and adapt. It is a later framework, not evidence of what OVH deployed in 2017. Applied carefully, it suggests that a network should be judged not only by prevention claims but by degradation boundaries, recovery authority, evidence preservation and adaptation after failure. [17]

The optical-transport architecture described by ITU recommendations provides vocabulary for layers, trail relationships and protection. It cannot reconstruct OVH's private topology from public facts. Its value here is to reinforce a general point: optical transport is a managed network with control, supervision and recovery functions, not merely passive glass. [18]

The accountability test is therefore independence, not duplication. An operator should be able to identify which elements are independent, which are intentionally shared and what happens when each shared element fails.

Saved configuration did not equal operational availability

OVH's incident account says saved configuration was restored. That detail illustrates a recurring problem in infrastructure assurance: backup presence is often treated as equivalent to recoverability.

A backup can exist and still fail to preserve service. It may be stored inside the same fault domain. Its replicas may be subject to the same faulty software. It may contain the same corrupt state. The system may not be able to select or load it automatically. Management access may be unavailable. Operators may need physical intervention. Restore procedures may be slow, ambiguous or untested under a common failure.

The Roubaix narrative indicates that engineers ultimately recovered saved configuration. That was a successful recovery action. It also shows that the running system did not maintain availability merely because a recoverable configuration existed. [1][4]

The relevant controls are therefore more precise than "a backup existed":

What exact object was copied: a complete database, a generated configuration, device state or transaction log?
Which process wrote each copy, and could one software defect corrupt all of them?
Were copies immutable or independently versioned?
Were they stored on physically and logically separate systems?
What consistency rule determined whether a copy was usable?
Could a known-good version be selected without the failed management component?
Was restoration automatic, operator-approved or dependent on local access?
How long had a full restore taken in the last test?
Did the restore recreate intended state or merely the last replicated state?
What evidence confirmed that every optical node and router-facing link had returned correctly?

The public material answers only part of the list. It shows that configuration restoration was possible. It does not show that recovery was independent, pre-authorized, regularly exercised or measured against a service objective.

This distinction should affect customer and board reporting. "Configuration was backed up" is an inventory statement. "A known-good configuration can be restored by an independent control path within a tested time, while preserving evidence and preventing reintroduction of the bad state" is an operational control statement.

Optical transport was part of the hosting service

Hosting accountability is often discussed at the server or data-centre level. The Roubaix failure shows why that boundary is too narrow. A server can remain powered and healthy while becoming unreachable because the optical transport connecting its site to external interconnection points has failed.

OVH's public peering material and current PeeringDB and RIPE records identify AS16276 and a substantial interconnection footprint. Those records are useful for network identity and operator attribution. They help a customer or investigator distinguish OVH's network from another provider and identify locations where interconnection may occur. They do not prove the operational state of a Roubaix circuit in 2017, the path used by a particular packet or the independence of any two customer services. [8][9][10][11]

This is a reality-layer distinction. A registry or directory can record who operates an autonomous system. A peering page can describe policy. A route collector can preserve selected BGP observations. None can make a failed optical system carry traffic.

Conversely, the optical failure did not erase network identity. The AS number, routes and external relationships remained meaningful evidence for diagnosing which network was expected to originate and carry traffic. Records and running infrastructure serve different accountability functions: records identify and preserve authority; running systems determine whether packets move.

Customers buying "redundant" hosting or multiple services from one provider need to ask whether their dependencies converge inside the provider. Two virtual machines in separate clusters may share the same site transport. Two services in different buildings may share a metro optical system. Two advertised paths may converge on one controller or configuration database. Product names and resource counts do not reveal these dependencies.

The Actility report provides an instructive external view. It said some services were unreachable while a SaaS offering hosted in another data centre was not affected. It also described reachability from some locations or networks but not others. [4] That pattern is consistent with dependency and path diversity, though the public data is not sufficient to map every route or service.

The customer lesson is not simply "use two providers." Multi-provider design can reduce concentration, but it creates its own DNS, routing, data consistency, security and operational complexity. The stronger requirement is to document dependency boundaries and test the failure that matters.

Impact must remain attached to observable evidence

OVH acknowledged direct consequences for other services and said that receiving customer email was especially difficult. The company apologized and said its teams remained mobilized. [1] Contemporary reporting described a significant customer impact across the Roubaix environment. [5][6][7]

The public sources do not provide a complete affected-customer count, a packet-loss time series, a route-by-route analysis, a service inventory, revenue impact or final SLA table. They do not establish that every workload in Roubaix was unreachable for the full optical interval. They also do not establish that a service was healthy merely because one public probe succeeded.

Different networks could have observed different behavior. Routing policies, cached DNS answers, existing sessions, alternate service locations and application retry logic can all affect visible impact. Some customers may have lost all access. Others may have reached a service through a remaining path or separate site. Email can queue and arrive later, making transport failure visible as delay rather than permanent loss.

The correct impact statement has three layers:

Provider-reported infrastructure impact: the Roubaix site lost the external optical links described in OVH's account.
Externally observed service impact: customers and at least one hosted provider reported partial or broad unreachability and specific service disruption.
Unknown complete impact: the public record does not enumerate every affected customer, flow, route, service or financial consequence.

Preserving these layers prevents both understatement and exaggeration. It does not minimize the event to say the complete impact is unknown. Losing a site's major optical links is inherently serious. It is also unnecessary to claim a universal outage when the evidence does not support one.

An accountable impact report would publish time-bounded reachability from independent networks, BGP session state, link availability, aggregate traffic, mail backlog, major service health, customer-ticket volume and restoration distributions. It would describe sampling limits and clocks. It would separate transport recovery from application recovery.

Control allocation: distributed responsibility is not absent responsibility

Network infrastructure crosses organizational boundaries. That can lead to a vague conclusion that responsibility was "shared." A better method allocates responsibility according to control.

OVH

OVH controlled the architecture of its Roubaix optical network, the relationship between physical paths and control systems, software deployment within its scope, configuration protection, monitoring, incident escalation, local intervention, restoration sequencing, customer communication and the decision to change the design.

That control creates specific obligations. OVH needed to identify common failure domains, stage software safely, preserve known-good configuration, maintain an independent diagnostic path, test restoration, set customer expectations and produce evidence that the corrected design contained recurrence.

OVH did not necessarily create the equipment software defect. That does not remove its architectural responsibility. Operators choose how one vendor defect can affect their service. They decide whether a defect reaches all redundant paths, whether rollback is possible and whether a failed controller can be bypassed.

The equipment manufacturer

The equipment manufacturer controlled product engineering, defect analysis, corrected software, vendor diagnostics and the disclosure available to OVH. The public record says OVH was investigating with the manufacturer and planned to upgrade the affected software. [1][4]

Without a public vendor report, this article cannot allocate the exact software error or assert whether the defect was previously known. The manufacturer nevertheless owned the product-side task of identifying why configuration disappeared or became unusable and demonstrating that corrected code prevented the failure.

Peers and transit providers

External network operators controlled their own links, BGP sessions, route preference, monitoring and escalation. They could observe loss of OVH reachability and adapt where alternative interconnection existed. They could not restore OVH's internal optical configuration.

BGP operational practices such as explicit import and export policy are essential at interdomain boundaries. RFC 7454 and RFC 8212 provide later or general routing safeguards, not a diagnosis of this optical event. They matter because restoration of light does not by itself prove correct route exchange. After interfaces return, explicit routing policy helps keep path recovery bounded. [15][16]

Customers

Customers controlled provider selection, service placement, external monitoring, DNS and application failover, data replication and their own incident communication. A customer that bought multiple products from OVH could ask for evidence that those products did not share the Roubaix optical failure domain.

Customers did not control OVH's internal optical-equipment state, configuration system or optical topology. It would be wrong to shift the provider's internal transport failure onto customers because they could have purchased more redundancy. Customer resilience and provider accountability are complementary, not substitutes.

Regulators, directories and observers

Registry, ASN and peering records support attribution and historical analysis. Measurement platforms can preserve selected routing observations. Standards bodies define useful design and operational principles. None operated the Roubaix system or could restore it.

This allocation avoids two errors. The first is treating a vendor bug as a complete excuse for provider failure. The second is assigning every consequence to OVH without recognizing customer and interconnection controls outside its boundary. Accountability follows the ability to prevent, detect, contain, recover and prove.

Detection and diagnosis are part of the control surface

The public record does not disclose the first internal alarm, its timestamp, severity, owner or diagnostic quality. We know that customers saw unreachability and that OVH mobilized teams. We do not know whether monitoring first identified optical transport loss, management failure, routing-session loss, traffic collapse or customer symptoms.

That missing evidence matters because detection design influences outage duration. An alarm that says "host unreachable" is less useful than a correlated record showing optical controller state, database health, card mode, management reachability, router interface state and external path loss.

Common-control failures can also compromise monitoring. If the monitoring card, management network or configuration service shares the same failure domain, the system may lose both service and explanatory telemetry. Engineers then face a dark failure: broad symptoms, incomplete remote access and pressure to restart equipment before volatile evidence is preserved.

An independent diagnostic path should not mean only a second interface on the same control plane. It should have separately powered and managed access, minimal dependencies, a known security boundary and the ability to capture state when the primary system is impaired.

The design also needs operational authority. Who may reset configuration? Which evidence must be captured first? Which known-good version may be loaded? When is local intervention required? How is the risk of a restart weighed against continued outage? A recovery plan that exists but cannot be authorized quickly is not an effective control.

The Roubaix narrative shows that restoring the optical system required more than observing that a backup existed. [4] That turns recovery time into an architectural parameter. If restoration depends on restarting and sequencing multiple components, those dependencies should appear in resilience tests and service objectives.

The announced repair separated defect correction from blast-radius control

OVH's formal action plan for Roubaix had two parts. It planned to investigate and fix the software defect with the equipment manufacturer through an upgrade. It also accelerated a project to distribute the optical multiplexing system across two separate systems to limit failure scope. [1]

That separation is technically important. Software correction addresses a known defect. Architectural separation addresses uncertainty, including the possibility of another defect or a different common-control failure.

A software patch cannot prove that an entire failure class has disappeared. It may fix one path through the code while leaving shared configuration, supervision or management dependencies unchanged. Conversely, splitting systems without correcting a known defect may produce two independently failing systems if the same software and state are deployed identically.

A strong remediation program would therefore test both dimensions.

Defect correction

Bind the vendor advisory or defect record to the affected software version.
Identify the exact failure condition, affected components and corrected behavior.
Stage the corrected image on a representative system.
Verify configuration migration, rollback and state preservation.
Exercise the condition that previously caused configuration loss.
Preserve logs showing that the failure no longer occurs.

Failure-domain separation

Document which optical paths terminate on each system.
Separate control, monitoring, configuration storage and management access where required.
Prevent one configuration transaction from disabling both systems.
Ensure a software rollout can be canaried and stopped before reaching every path.
Prove that loss of one system leaves enough external capacity and reachability for the defined service objective.
Test traffic movement and routing convergence under the reduced topology.

Recovery evidence

Restore a known-good configuration without relying on the failed control component.
Measure detection, decision, restore, link return and route convergence times.
Preserve before-and-after checksums and topology state.
Compare internal link state with independent external probes.
Record residual errors rather than declaring all-clear at the first successful ping.

OVH's public statement documents an intention to make the split. The sources frozen for this article do not provide a completion date, implementation diagram, independent test or later recurrence exercise. The responsible conclusion is that the announced direction addressed the right control distinction, while its effectiveness remains unproven in this record.

How to test redundancy independence

An operator can turn the incident into a repeatable audit by constructing a failure-domain matrix. Each customer-relevant path is mapped against physical route, optical nodes, line cards, chassis, supervisory systems, configuration stores, software release, management network, power, timing, operator group and restore authority.

The matrix should answer a simple question for every pair of "redundant" paths: which components can still remove both?

That analysis often reveals dependencies hidden by topology diagrams. Separate fibres may enter the same building. Separate chassis may use one controller. Separate controllers may share one database cluster. Separate software instances may receive one bad configuration from a common automation pipeline. Separate data centres may depend on one DNS, identity or network-control service.

The matrix is only a hypothesis until tested. Useful tests include:

remove one physical fibre and verify automatic optical protection;
remove one chassis and verify that capacity remains within the declared objective;
isolate one controller and verify that the other system continues without unsafe state convergence;
corrupt or withhold one configuration replica and verify safe selection;
deny the primary management path and perform diagnosis over the independent path;
deploy a deliberately rejected software image to a canary and verify rollout containment;
restore a known-good configuration under time pressure;
measure BGP and data-plane convergence from independent networks;
verify that customer-facing status reflects observed service, not only device health;
retain evidence sufficient for an outside reviewer to reproduce the conclusion.

The pass condition should be defined before the test. "Traffic recovered" is too vague. A useful condition states minimum capacity, maximum reachability loss, permitted packet loss, convergence time, service classes, external vantage points and evidence-retention requirements.

Tests should also account for maintenance. Many common failures occur during upgrades or configuration changes, when redundant systems are intentionally aligned. A design that survives a random card loss may fail when a common automation job pushes the same bad state to both sides.

Independent operation does not require every component to be different. Complete heterogeneity can increase complexity and error. It requires that shared dependencies be explicit, bounded and matched to tested recovery. The goal is not diversity theatre. It is evidence that one plausible failure cannot silently defeat every path advertised as redundant.

Counterfactual controls clarify the first missed opportunities

Counterfactual analysis should not pretend that one control would certainly have prevented the event. It asks where observable controls could have changed the sequence.

If the software defect had been caught before deployment

A representative staging environment, canary rollout or vendor defect test might have exposed the failing state before it reached the production system. Public evidence does not tell us whether the defect required a rare sequence impossible to reproduce in staging. This counterfactual is therefore possible, not proven.

If optical systems had independent control state

If two path groups had separate configuration authority and supervisory systems, loss of one configuration database might have left the other group operational. OVH's announced system split suggests the company saw value in reducing the common failure scope. The exact pre-incident topology is not public, so the magnitude of the counterfactual cannot be calculated.

If configuration restoration had an independent path

Known-good, immutable configuration and an out-of-band restore mechanism might have reduced diagnosis and recovery time. The incident account says saved configuration was ultimately restored. It does not show whether an independent restore path existed, how it was tested or how much of the recovery depended on the affected control environment.

If customers had independent hosting paths

A customer using another data centre or provider could have avoided some service impact. Actility reported that a SaaS offering in another data centre was not affected. [4] That observation supports dependency diversification, but it does not prove that every multi-site design would have succeeded.

If external reachability evidence had been integrated

Independent probes and routing observations could have helped distinguish local device recovery from customer reachability. They would not have restored optical links. Their value would have been faster diagnosis, scoped communication and stronger proof of closeout.

The first missed opportunity cannot be named conclusively from public evidence. It may have been software assurance, architecture, configuration-state protection, monitoring, recovery design or a combination. A private incident record should identify the earliest control that both had authority and could reasonably have acted.

Evidence that would change the conclusion

The conclusion is deliberately falsifiable. It should change if stronger evidence becomes available.

Device and controller logs could show that configuration loss was a consequence rather than a cause. A vendor report could identify a specific hardware or software condition. Topology records could show that the optical paths were more independent than the public account implies, or that another shared component removed them. Configuration histories could show that a human change, automation job or state transition triggered the event.

Customer and measurement data could revise the impact boundary. A complete reachability analysis might show near-universal Roubaix isolation, or substantial remaining connectivity. BGP collector data could establish which external sessions and paths changed, but it would still need data-plane measurements to support forwarding conclusions.

Remediation evidence could strengthen or weaken the accountability finding. A completed system split, independently tested under controller and configuration failures, would show that OVH converted the lesson into containment. A later test showing that both systems still shared one failure domain would show that duplication had not delivered independence.

The article does not require those records to state that a serious network failure occurred. It requires them to make the repair claim complete.

Conclusion

OVH's Roubaix outage was not a lesson that redundancy is futile. It was a lesson that redundancy must be described in terms of the failures it can contain.

The provider had redundant optical connectivity and saved configuration. Those controls addressed real risks. They did not prevent a shared optical-control failure from removing Roubaix connectivity, and restoration still depended on recovering the configuration.

OVH's later action plan recognized the two layers of the problem: fix the software defect and divide the optical multiplexing system so that one failure has a smaller scope. That was the right conceptual separation. Public evidence in this source set does not prove completion or recurrence resistance.

For hosting and network operators, the accountability standard is concrete. Name the protected failure classes. Map shared control domains. Preserve independent diagnosis and known-good recovery. Test one system's loss while the other carries traffic. Measure reachability from outside the provider. Publish enough evidence to distinguish component duplication from operational independence.

Network identity records and topology inventories can show who operates the infrastructure and what is supposed to exist. Only running-code evidence, observed reachability and tested restoration show whether the infrastructure continues to work.

OVH's 2017 Optical-Network Failure Made Redundancy Independence an Accountability Test