AT&T's 1990 Long-Distance Outage Made Signaling-System Change Iso

Summary

At about 2:30 p.m. EST on 15 January 1990, a New York 4ESS toll switch encountered a minor hardware problem and entered a normal recovery sequence that suspended new call processing for four to six seconds. The hardware event initiated the sequence but was not the software defect that allowed it to spread. [1]
When the New York switch returned to service, it sent Initial Address Messages, or IAMs. AT&T reported that two IAMs arriving at a receiving Direct Link Node processor within one hundredth of a second could damage data while that processor updated its view of the recovering switch. [1]
A duplicated DLN processor did not supply an independent software boundary. After one processor reinitialized, its mate could receive the same message pattern during the same vulnerable state transition, temporarily cutting the switch off from CCS7 signaling. [1]
Recovery announcements, renewed call setup and continuing signaling load allowed the condition to recur at neighboring switches. The network’s recovery behavior therefore became part of the propagation path rather than a guaranteed route back to stability. [1][10][11]
AT&T said the vulnerable code entered the 4ESS fleet in a mid-December update intended to provide faster access to backup signaling. A contemporary Telephony account reported that the code was loaded in front-end processors across all 114 4ESS systems; that does not mean all 114 failed simultaneously. [1][2]
AT&T reported stabilizing the network by suspending traffic on backup signaling links, reducing the load reaching affected processors. It said the last link was cleared at 11:30 p.m. EST, followed by rollback, laboratory reproduction, correction and testing. [1]
Contemporary estimates must remain separate: roughly half of attempted long-distance calls reportedly failed during substantial periods, while UPI reported an estimate of 50 million blocked calls. Later accounts differ, and the available record does not establish one audited total or a confirmed dollar loss. [3][4][5][6][9]
The central accountability test is operational: whether common signaling software could be tested under production-equivalent transitions, released through a bounded canary, stopped on measured warning conditions, isolated during failure and rolled back with verifiable evidence.
Redundancy is not continuity proof when duplicate processors, alternate signaling paths and a large switch fleet share the same code, state-transition semantics and triggering message pattern.

The event boundary is 15 January 1990

This analysis is confined to the disruption in the American Telephone & Telegraph Company’s long-distance network on 15 January 1990 and the immediate stabilization, rollback and technical investigation that followed. The historical operator should not be treated as though current AT&T Inc. already had its present corporate form. The event also must remain separate from AT&T’s 2024 wireless configuration outage, later power and fibre incidents, 911 failures, frame-relay problems and malicious uses of SS7. Those events involve different systems, evidence and accountability questions.

The evidence supports a bounded causal account, not a complete forensic reconstruction. AT&T’s published technical explanation describes a hardware-triggered recovery, a timing-sensitive software vulnerability in signaling processors, network-wide propagation under message load, operational stabilization and subsequent rollback and laboratory reproduction. Contemporary reporting supplies service-impact estimates and operational observations, while later research provides a framework for understanding control-plane messages that repeatedly expose latent implementation defects. [1][3][4][5][10][11]

No source in this set establishes an attack, virus, sabotage, anomalous demand, regulatory violation or individual act of negligence. AT&T reportedly characterized traffic volume as lower than on a normal Monday, but that observation does not by itself prove every alternative cause false. It does reinforce the narrower conclusion that the reported cascade depended on recovery state, shared software and signaling-message timing rather than extraordinary customer demand. [1][2]

4ESS switching and the CCS7 control layer

A 4ESS was a toll switch within AT&T’s long-distance network. It participated in carrying and controlling calls between parts of that network, but the call-bearing path and the common-channel signaling system should not be described as the same thing. CCS7 carried control information used to establish calls, communicate availability and coordinate network behavior. A bearer path could physically exist while failures in signaling prevented the network from using it to complete calls.

The distinction matters because the cascade was centered on control state. Initial Address Messages were call-setup messages. Status and recovery information told connected switches whether a destination switch should receive new traffic. Direct Link Node processors handled access to the signaling network and maintained information needed to interpret that changing state. A failure in this layer could therefore make a substantial portion of the long-distance network operationally unavailable without requiring every voice circuit or every switch chassis to suffer a physical failure. [1]

ITU-T’s Signalling System No. 7 framework describes a common-channel architecture in which a Message Transfer Part conveys signaling among user functions. Its associated recommendations address functional structure, signaling-network functions, performance, monitoring and measurement. They help explain the architectural role of status, routing, recovery and call-establishment messages. They do not, on this evidence, prove that AT&T breached a particular recommendation in 1990. [14][15][16][17][18][19]

NTIA and GAO materials similarly provide broader context for restoration, network reliability and the public consequences of telecommunications failures. They are useful because continuity depends on knowing which control functions are available, how failure is detected and how service is restored. They should not be converted into retroactive findings about AT&T’s compliance, liability or internal decisions. [12][13][20]

A minor hardware event opened the failure path

AT&T’s account places the initiating event at about 2:30 p.m. EST. A New York 4ESS experienced what the company described as a minor hardware problem associated with a trunk interface. Normal fault-recovery logic responded by suspending new call processing and informing connected switches not to send new traffic. The New York recovery began at about 2:30 p.m. EST and took four to six seconds before the cascade condition appeared as the switch returned to service. [1]

That sequence defines the first important distinction. The hardware problem was the trigger: it moved the switch into a recovery transition. It was not the reported reason that other switches later entered repeated signaling failures. A sound analysis must therefore resist collapsing “what happened first” into “what made the event nationally consequential.”

The recovery routine initially did what a protective mechanism was expected to do. It paused new work, allowed the affected switch to recover and communicated unavailability to its peers. The critical exposure came during the transition back to normal processing. That is a class of risk easily missed when testing concentrates on steady states—fully available or fully unavailable—without applying realistic traffic to the narrow interval between them.

How IAM timing exposed the DLN state vulnerability

When the New York switch resumed call processing, neighboring switches needed to recognize that it was available again. The resumed flow included IAMs, the messages used to initiate call setup. A receiving switch’s DLN processor used incoming information to update its status map—the operational view through which it understood that New York had returned to service. [1]

AT&T reported that the update left the DLN processor vulnerable for several seconds. If two IAMs arrived within one hundredth of a second during that interval, the closely spaced messages could damage data and force the processor to reinitialize. The one-hundredth-of-a-second figure is AT&T’s reported timing condition; it should not be expanded into a claim that the public record contains the complete instruction trace, data structure or source code needed to reproduce the defect independently. [1]

The accessible mechanism is therefore a collision between valid traffic and an unsafe transition. Neither IAM had to be malicious or malformed. The problem emerged because ordinary call-setup messages arrived at a particular spacing while the receiving processor was changing its operational view. The message sequence exposed a latent defect in running software at precisely the moment that the network was trying to record and act on a state transfer from unavailable to available.

This is why the incident is better understood as a state-integrity problem than as a simple excess-traffic event. A processor must not only receive messages; it must record the network’s current state accurately while messages continue to arrive. If the state-update operation and the live message stream are not safely coordinated, a valid input can produce corrupt internal state, a reset or an incorrect availability decision.

Some later software-engineering narratives describe the defect in terms of a misplaced break statement. That description is secondary and is not equivalent to primary-source verification of the code. The available sources do not expose the relevant source tree, compiled program, surrounding control flow or exact corrupted structure. The defensible account is the mechanism AT&T reported: closely spaced IAMs encountered a vulnerable DLN status update and caused processor reinitialization. [1][8][9]

Duplicate processors shared the same failure condition

The receiving signaling function had duplicated processors. When one DLN processor failed and began reinitializing, its mate could assume the workload. In a hardware-centered model, that transfer can look like redundancy: one unit fails, so the other keeps the function available. AT&T’s explanation shows why that description was incomplete. [1]

The mate ran the same vulnerable logic, participated in the same recovery state and could receive the same kind of closely spaced IAM pair. If another pair arrived while the mate was updating its status, it too could be forced to reinitialize. With both processors temporarily unavailable, the switch could be isolated from the CCS7 signaling network even if other switching hardware remained operable. [1]

The two processors were duplicated components but not independent failure domains. Their hardware may have been separate, yet their software, input semantics and transition timing were correlated. A failover that transfers the same message stream to the same code during the same vulnerable state can reproduce the failure instead of containing it.

That distinction is central to infrastructure accountability. Redundancy should be evaluated against the failure being controlled. Duplicate hardware protects against some component faults. It does not necessarily protect against a common software defect or a valid message pattern that drives both units through the same unsafe transition. Independence must be demonstrated at the level of code, state, input and recovery behavior—not inferred from a component count.

Recovery announcements became propagation signals

Once a switch lost its signaling processors, it communicated an unavailable condition to connected switches. After its processors reinitialized, the switch returned to service and signaling resumed. Neighbors then had to update their own views and process renewed call-setup traffic. Under continuing message pressure, they could encounter the same timing-sensitive DLN condition. [1]

The cascade was therefore not a single signal radiating mechanically from New York. It was a repeating cycle of local failure, unavailability, reinitialization, recovery announcement and renewed traffic. Each return-to-service transition created another opportunity for closely spaced IAMs to reach a vulnerable processor. AT&T described the resulting failures as random in their occurrence under ongoing traffic, but the underlying vulnerability was shared. [1]

Later research on control-plane “poison messages” provides a useful analytical frame. In that work, an otherwise legitimate control message can expose a latent implementation defect, and normal management behavior can repeatedly deliver or recreate the triggering condition. The term does not imply hostile content in this event. Its value is in showing how a control plane may perpetuate instability when recovery messages, state changes and message replay remain coupled to the same vulnerable implementation. [10][11]

The feedback path also explains why restoring one switch was not necessarily sufficient. Local recovery changed the inputs seen elsewhere. A recovered switch did not return to an isolated laboratory state; it returned to a live signaling environment containing queued demand, availability changes and neighboring systems running the same software. Recovery had to be judged by whether the network remained stable after traffic and state exchange resumed.

The mid-December change created a broad common boundary

AT&T said the flaw had entered the 4ESS fleet through a software update installed in mid-December. The update was intended to let switches reach a backup signaling network faster. A contemporary Telephony account reproduced in the RISKS archive reported that the new code was loaded into front-end processors in all 114 4ESS systems. Both points should remain attributed to AT&T and contemporary reporting rather than presented as findings from an independent source-code or configuration audit. [1][2]

The 114-switch figure defines an inventory boundary, not a simultaneous-failure count. The public record supports saying that a common software condition was widely deployed. It does not show that all 114 switches failed at once, failed for the same duration or produced identical effects for every caller. A cascade can be national in consequence while remaining staggered and uneven in its component-level timeline.

The change’s intended purpose sharpens the control issue. Software meant to improve access to backup signaling became the shared source of vulnerability. That does not make backup capability inherently harmful. It shows that a continuity feature can widen a common failure domain when its activation and transition logic are deployed uniformly without sufficiently independent containment.

A fleet-wide change creates two accountability questions. The first is whether the code is correct under production conditions. The second is whether a defect can remain bounded while the answer is still uncertain. Even rigorous pre-release testing cannot enumerate every timing combination. Change isolation, staged exposure, stop criteria and rollback therefore remain necessary when laboratory confidence is high.

The sources do not disclose the rollout order, how quickly the 114 systems received the update, whether a canary subset existed, which measurements governed expansion or whether a stop threshold was defined. Those are unknowns, not proof that no controls existed. Their absence from the public record nevertheless limits any claim that fleet risk was demonstrably bounded before full exposure.

Observation, diagnosis and stabilization were different phases

The incident response should be divided into what engineers observed, what they diagnosed, how they stabilized the live network and what they later verified. Treating those phases as interchangeable would overstate what was known at each point.

The initial observation was operational: switches were generating errors, entering recovery and disrupting long-distance call completion. AT&T said engineers first applied standard procedures but found them inadequate. They then examined patterns in error messages and switch behavior and brought technical support and Bell Labs personnel into the investigation. [1]

Diagnosis required recognizing that apparently separate switch failures were linked by signaling behavior and shared software. The initiating hardware problem alone could not explain the continuing network-wide instability. Nor would merely restarting processors remove the condition, because reinitialization and resumed message flow could place the next processor or neighbor into the same vulnerable transition.

Stabilization came before a complete software correction. AT&T reported temporarily suspending signaling traffic on backup links, thereby reducing the load of messages reaching affected processors. This was a control-plane intervention: it altered the environment in which the defect was being triggered. The company said the last such link was cleared at 11:30 p.m. EST. [1]

That action matters because it demonstrates the practical value of isolation even when the root mechanism is not yet fully reproduced. An operator needs a way to lower, block or redirect suspect signaling traffic without losing all visibility into recovery. Stabilization evidence should show which links were changed, when message rates fell, which processors stopped reinitializing and whether call completion improved. The public sources establish the broad action and endpoint, not that complete measurement record.

Rollback and laboratory reproduction closed different questions

On Tuesday, AT&T removed the faulty update and temporarily returned the network to the previous program. The company also said it reproduced the problem in a laboratory, corrected the flaw, tested the revised software and then restored the backup links. [1]

Rollback supplied evidence that the prior version was a usable recovery point. Laboratory reproduction supplied evidence that the reported software condition could be triggered under controlled circumstances. Correction and testing addressed the identified flaw. These steps reinforce one another, but none should be used as a substitute for the others.

A return to stable service after rollback supports the relationship between the update and the failure. It does not reveal the full defective code or prove that every affected switch followed the same timeline. Laboratory reproduction supports the timing-sensitive mechanism. It does not show the public a complete test matrix, production message trace or independent verification of all remediation measures.

AT&T said the update had been rigorously tested in laboratory environments and that the exact event combination had not been predicted. That statement must remain attributed. The public record does not expose which state transitions, message spacings, processor-failover sequences or signaling loads were tested before deployment. [1][2]

The deeper lesson is therefore not that laboratory testing was absent. It is that evidence of code-path testing alone cannot establish continuity. The critical test involved a live transition: one switch returning from unavailability, neighboring processors updating status, closely spaced IAMs arriving, duplicated processors transferring work and the network continuing to exchange recovery information.

Impact estimates cannot be compressed into one number

Contemporary reporting described a severe but uneven long-distance disruption. AT&T and major newspapers reported that roughly half of attempted calls failed during substantial periods. The phrasing matters: “about half” is an estimate, “attempted calls” defines the denominator, and “during substantial periods” does not mean a uniform failure rate across every location and every hour. [3][4][6][7]

UPI reported an estimate of 50 million blocked calls and effects on 800-number and computer-line services. That figure should stand as UPI’s contemporary estimate rather than be merged with later accounts into a supposedly reconciled total. [5]

Later narratives report different or higher totals, illustrating how retrospective figures can diverge according to time window, source, definition and repetition. A blocked attempt is not automatically a unique customer, a permanently lost call or a measured economic transaction. Repeated dialing may create multiple attempts, while some customers may have used alternate routes or tried again after service recovered. [8][9]

The available evidence does not establish uniform geographic availability, a complete list of affected customers, the exact number of emergency, financial or data transactions disrupted, or an independently audited total. It also does not support a confirmed dollar-loss figure. Economic consequences may have existed, but assigning a precise amount without a defined method and primary evidence would manufacture certainty.

Responsible reporting should therefore preserve parallel measures: observed call-completion performance, operator estimates, press estimates, affected service categories and explicitly unknown consequences. Severity does not need false precision. A national control-plane cascade that caused approximately half of attempted calls to fail during substantial periods is already a material continuity event.

Trigger, defect, propagation and mitigation form one causal ladder

The incident becomes easier to govern when its causal stages are kept separate.

Trigger: A minor hardware problem at a New York 4ESS initiated a normal four-to-six-second recovery sequence.
Latent defect: A mid-December software update left DLN status-update processing vulnerable during the return-to-service transition.
Propagation condition: Closely spaced IAMs—reported by AT&T as within one hundredth of a second—could force a processor to reinitialize; its duplicated mate and neighboring switches could encounter the same condition.
Mitigation: Signaling traffic on backup links was suspended to reduce load and stabilize processor behavior.
Removal and verification: AT&T rolled back the update, reproduced the flaw in a laboratory, corrected it and tested the change before restoring the affected capability. [1][2]

This ladder prevents three distortions. First, it avoids calling the initiating hardware fault the root cause of the national cascade. Second, it avoids treating message volume as an independent cause when timing, state and vulnerable code were necessary. Third, it avoids describing rollback as the initial stabilization action when signaling-load reduction preceded full software removal.

It also locates accountability without assigning unsupported personal blame. The operator controlled the common software release, switch inventory, signaling relationships, recovery behavior, monitoring and rollback path. Standards bodies defined architectural context, but the sources do not establish that a standard caused the defect. Customers produced ordinary call demand, not evidence of misuse. Individual engineers and operators should not be blamed without records showing their authority, knowledge and decisions.

Operational accountability asks whether the organization controlling the infrastructure could demonstrate that the transition was safe, observe when it was not, prevent one failure domain from reaching the fleet and reverse the change promptly. It is a control test grounded in running behavior, not a judgment based solely on intent or system labels.

Production-equivalent testing must exercise transitions

A conventional test may verify that a switch works before a fault and works after a restart. That is limited public evidence for this failure mode. The vulnerable interval sat between those states, while status maps were changing and live IAMs continued to arrive. Production-equivalent testing must therefore exercise transitions under realistic concurrency, message spacing, failover and network feedback.

A suitable transition scenario would begin with an available switch under representative signaling load. It would introduce the bounded hardware-recovery event, verify the temporary unavailability announcement, hold new call processing for the expected interval and then restore service. During the status-update window, the test would deliver closely spaced IAMs across a range that includes AT&T’s reported one-hundredth-of-a-second condition. It would observe both DLN processors, not merely the first unit to receive the messages. [1]

The test must then continue beyond the first restart. It should determine whether the mate takes traffic safely, whether both processors share corrupted state, whether a recovered switch’s announcements cause neighboring switches to enter the same transition and whether messages queued during unavailability create a burst when service resumes. A pass is sustained network stability after control traffic returns—not simply successful process reinitialization.

Message replay is valuable when it preserves timing and state context. A list of message types without their spacing, ordering, switch status and processor ownership may omit the condition that made the sequence dangerous. Replay evidence should therefore bind each input to a software version, pre-transition state, timing reference, receiving processor, observed state change and resulting output.

Testing also needs fault combinations. The exact 1990 sequence joined a minor hardware recovery with a latent software flaw and ordinary signaling traffic. Each element might pass an isolated test. Their interaction did not. Production-equivalent testing should combine component failover, status transfer, resumed call setup, duplicate-processor handoff and neighboring-node behavior.

AT&T’s statement that the update was rigorously tested but the exact combination had not been predicted illustrates the limit of relying only on anticipated scenarios. [1] The answer is not an impossible promise to predict every event. It is a test program that emphasizes transition classes, applies structured perturbations and treats unexpected reinitialization, inconsistent status or correlated failover as a release-blocking result.

A failure threshold should be explicit. Examples at the control level include any unexplained DLN reinitialization, disagreement between duplicated processors, loss of signaling reachability during a permitted single failure, propagation to a neighboring canary switch, or inability to restore the previous version within the tested recovery objective. Exact numeric thresholds beyond the documented timing condition would require operator measurements not available here.

Canary exposure needs enforceable stop conditions

A canary is a deliberately bounded first exposure to a new software version. For a switch fleet, it is not simply “install on one switch first.” The selected scope must reveal the relevant behavior while limiting the number of peers, routes and customers that can be affected.

For this change, a meaningful canary would need representative CCS7 relationships, dual-DLN behavior, access to primary and backup signaling, real return-to-service transitions and controlled production load. A laboratory-only node or an operational node that never exercises recovery could produce confidence without testing the risky state.

Expansion should depend on measured evidence. The release record should identify the software version, canary switches, adjacent signaling relationships, observation window, expected transition behavior and authority to proceed. Stop conditions should include processor reinitialization linked to valid messages, inconsistent status maps, repeated availability changes, unexpected failover coupling, loss of signaling access or abnormal propagation to a neighboring switch.

A stop condition has little value unless it blocks further release. The responsible owner must have both the authority and the technical means to halt deployment, isolate signaling exposure and preserve the prior version. If the expansion process continues by default while evidence is reviewed later, the canary is observational rather than protective.

The public record does not reveal whether AT&T used a canary, what version inventory existed or what stopped the mid-December rollout. That missing evidence cannot prove the absence of staged deployment. It does mean that the breadth of the common software boundary cannot be evaluated against a documented public release gate.

Fleet inventory is equally important after deployment. During an incident, responders need to know which switches run the suspect version, which have entered recovery, which processors have reinitialized and which signaling links remain active. A nominal count of 114 systems is not enough; control depends on current state attached to each system. [2]

Signaling isolation and message-level observability

AT&T’s reported stabilization action—temporarily suspending traffic on backup signaling links—shows that signaling isolation was operationally important. Reducing the input load gave affected processors room to recover without immediately encountering the same triggering pressure. [1]

Isolation controls should be narrow enough to arrest propagation while preserving useful evidence and recovery options. Depending on architecture, that may mean rate-limiting a message class, blocking a suspect relationship, holding return-to-service traffic, separating a canary from fleet peers or temporarily disabling an alternate path. The sources do not establish which of these controls existed beyond the reported backup-link action, so they are control objectives rather than claims about AT&T’s configuration.

Message-level observability must distinguish a trigger from a cascade. At minimum, evidence should identify the initiating hardware state, the unavailability notice, the start and completion of recovery, IAM arrival timing, DLN status-map transition, processor reinitialization, mate takeover, signaling isolation and subsequent restoration.

Aggregate traffic graphs alone would be inadequate. A national drop in call completion indicates severity but does not explain why two valid messages affected one processor during a narrow transition. Conversely, a low-level processor trace without network-state context might show a restart but miss how recovery announcements reproduced the condition elsewhere.

Time alignment is especially important. The one-hundredth-of-a-second condition reported by AT&T places analytical weight on precise ordering. Evidence should use a consistent clock reference and preserve enough resolution to determine whether messages arrived before, during or after the vulnerable state update. [1] The public record does not establish the available clock accuracy, telemetry retention or message-trace completeness in 1990.

Alert ownership also remains unknown. A useful control assigns responsibility for acknowledging correlated processor resets, recognizing a fleet pattern, declaring a release stop, isolating signaling and authorizing rollback. The aim is not to identify an individual to blame after the fact. It is to ensure that an observed unsafe transition results in a timely, blockable action.

Backup signaling can share the defect it is meant to overcome

A backup path creates continuity only when it remains sufficiently independent of the primary failure. AT&T said the mid-December update was intended to provide faster access to a backup signaling network, yet that software introduced the vulnerability implicated in the cascade. [1]

The paradox is operational rather than rhetorical. If primary and backup signaling access depend on the same new code, the same DLN state machine and the same message timing, the alternate path may preserve physical connectivity while sharing the software failure domain. Faster access to backup signaling then cannot guarantee useful continuity.

The same principle applies to duplicated processors. Hardware duplication, link diversity and fleet scale are valuable, but they address different failure classes. Independence must be evaluated across software version, state transition, input source, control authority and restoration method. If all copies enter the same unsafe transition, the label “redundant” records topology but not resilience.

Rollback must also remain independent. The previous program, installation mechanism and authorization path should be usable when the new signaling logic is unstable. AT&T’s reported return to the prior program demonstrates the role of a preserved recovery version, although the public sources do not disclose the rollback timing for each switch or the internal authorization sequence. [1]

An operational evidence register for fleet state

Continuity decisions require an accurate, time-indexed record connecting software and network state. An operational evidence register should link five core elements for each switch:

the installed software version and its provenance;
the switch’s availability and recovery state;
the health and active/standby status of each DLN processor;
the status of primary, backup and isolated signaling relationships; and
the release, isolation or rollback authorization currently in force.

Those fields must be linked rather than stored as independent snapshots. A version record without processor state cannot show whether the suspect code was active during a reset. A processor alarm without signaling status cannot show whether the switch was isolated or still exposing neighbors. A release approval without a measured canary result cannot demonstrate why expansion was authorized.

The register should capture transitions as events: switch unavailable, recovery begun, status update active, IAM burst observed, processor reinitializing, mate active, signaling relationship suspended, previous software restored and stable operation verified. Each event should carry a timestamp, source system, confidence level and responsible control owner.

Release authorization should be conditional. A canary may advance only when required artifacts are attached: transition-test results, message-replay evidence, processor-consistency checks, rollback exercise and an observation window without threshold breaches. A failure artifact should automatically block expansion until an authorized review records disposition and supporting evidence.

During an incident, the same record becomes the basis for diagnosis. Responders can compare affected and unaffected switches by version, state and signaling exposure. They can identify whether failures track a software boundary, a message source, a processor role or a recovery phase. That is more useful than treating “the network” as a single binary entity.

After restoration, public reporting can derive bounded statements from measured records: how many switches entered recovery, how long signaling relationships were isolated, when the previous version was restored and how call-completion performance changed. The 1990 public sources do not provide that complete register. Its absence is one reason impact and per-switch timing must remain qualified.

What the public evidence does not resolve

The sources do not expose the defective source code, compiler behavior or exact data structure affected by the timing condition. They do not provide the complete pre-production test matrix, the change-review record or a production trace showing every IAM and DLN transition.

The rollout sequence across the 4ESS fleet remains unknown. So do any canary boundaries, observation windows, stop criteria, version-inventory controls and release authorizations. The contemporary 114-switch report establishes the stated deployment scope, not the order or governance of deployment. [2]

There is no complete per-switch failure and recovery timeline. Public accounts do not establish which switch failed at each moment, how long each processor remained unavailable, how signaling isolation varied by region or whether every remediation step had identical effects.

Customer impact is similarly incomplete. The record supports substantial long-distance call failures and reports effects on 800-number and computer-line services, but not a reconciled customer list, precise emergency-service effect, audited transaction loss or confirmed economic total. [3][4][5][6]

Internal incident-command decisions, alert acknowledgement and rollback-authorization times are not public here. The reason for the initiating trunk-interface hardware fault also remains unresolved in this source set. There is no basis for converting those gaps into allegations of misconduct, malicious compromise or individual fault.

Finally, successful immediate remediation does not prove that every later process change was effective. Evidence of revised testing, release isolation and rollback exercises would be needed to evaluate long-term control improvements. The appropriate response to each unknown is a bounded evidence request, not speculation.

Control-and-evidence matrix

Control	Owner	Test	Artifact	Failure threshold
Production-equivalent recovery testing	Switch-software release owner	Recover a 4ESS-equivalent node under representative CCS7 load while replaying closely spaced IAMs	Version-bound trace of switch state, IAM timing and DLN transitions	Any unexplained reset, state corruption or loss of signaling reachability
Correlated-processor validation	Signaling-platform owner	Exercise primary and duplicate DLN processors through the same recovery and message sequence	Processor-role timeline and state-consistency report	Both processors enter the same unsafe transition or cannot preserve service
Bounded canary	Fleet change owner	Deploy to a representative, isolated subset before expansion	Canary inventory, peer map, observation window and approval record	Propagation beyond the canary or any release-gate breach
Enforceable stop gate	Release authority	Inject a threshold breach during rehearsal	Recorded automatic or authorized deployment halt	Expansion continues after a stop condition
Signaling isolation	Network operations	Limit or suspend triggering traffic while retaining monitoring	Link-state history, message-rate trace and restoration record	Isolation cannot arrest propagation or destroys recovery visibility
Message replay and observability	Reliability engineering	Reconstruct the transition with preserved order and timing	Timestamped message/state trace tied to software version	Timing or receiving-state context is missing
Rollback	Release and operations owners	Restore the preserved prior version under incident conditions	Per-switch rollback record and post-rollback stability evidence	Prior version cannot be restored within the tested objective
Impact reporting	Service assurance	Reconcile measured completion rates with attributed estimates	Time-bounded metrics with definitions and uncertainty	Estimates are presented as audited totals or unsupported financial loss

A bounded accountability test

The 15 January 1990 outage was not made nationally significant by a minor hardware recovery alone. Its scale arose from the interaction of a timing-sensitive DLN defect, valid IAM traffic, coupled duplicate processors, repeated CCS7 recovery transitions and a common software boundary across the 4ESS fleet. [1][2]

The first accountability test is measured transition behavior. Can the operator show what happens while a switch moves from unavailable to available under real signaling load—not only before and after that movement? The evidence must cover message timing, state updates, processor handoff and effects on neighboring switches.

The second is failure-domain isolation. Duplicate processors and backup signaling should be treated as independent only when tests demonstrate that they do not share the relevant code, state or input failure. A large fleet does not improve continuity when one release gives every node the same vulnerable transition.

The third is a blockable release gate. A canary must expose the risky behavior, and its warning conditions must stop expansion. Version inventory, processor state, signaling status and release authority must remain accurately connected so responders can identify the shared boundary while the event is unfolding.

The fourth is exercised rollback. A preserved prior version is useful only if operators can restore it while signaling is unstable, verify that the defect no longer reproduces and control the later restoration of isolated links. AT&T’s reported rollback and laboratory reproduction illustrate the distinct evidence required for removal, diagnosis and correction. [1]

The final test is transparent evidence. Impact estimates should remain attributed, unknowns should remain visible, and technical confidence should rest on measured running behavior rather than labels such as “backup,” “duplicate” or “rigorously tested.” Continuity is demonstrated when transitions are safe, failure domains are isolated, release gates can halt deployment, rollback works under stress and the evidence connecting those controls can withstand review.

AT&T's 1990 Long-Distance Outage Made Signaling-System Change Isolation an Accountability Test