Summary

  • The FCC order says a Verizon Wireless outage on 21 December 2022 affected wireless VoLTE 911 traffic in Alabama, Florida, Georgia, North Carolina, South Carolina and Tennessee. VoLTE means voice calls carried over a 4G LTE network.
  • The FCC says the outage lasted one hour and forty-four minutes, or 104 minutes, and prevented hundreds of 911 calls from completing through Verizon Wireless's network. The public sources do not give an exact call count, unique-caller count or confirmed individual harm.
  • The FCC record links the December event to a similar outage in October 2022. It says Verizon conducted a root-cause analysis and introduced audits and technical updates intended to prevent recurrence of configuration and one-way-audio issues after the October event.
  • The consent decree identifies the December trigger as an employee's reapplication of a known flawed security-policy update file. It also says the file remained in the available inventory, naming conventions were insufficient, then-current procedures were not followed and required additional oversight was not applied.
  • The December event did not include the earlier one-way-audio issue. The two events should therefore not be described as technically identical.
  • On 25 June 2024, the FCC Enforcement Bureau adopted a consent decree—a negotiated settlement—and terminated its investigation. Verizon admitted the paragraph 4 facts for purposes of the decree and FCC civil enforcement. The settlement required a $1.05 million civil penalty, a monetary agency-enforcement payment rather than a criminal sentence, and a compliance plan; it was not a court judgment.
  • The central control lesson is that an announced corrective action is not yet a tested safeguard. A working safeguard makes a known bad file unavailable, blocks inappropriate reapplication, tests significant changes against representative conditions and leaves evidence that those barriers worked.

What happened

On 21 December 2022, people in six US states encountered a failure in a service most users expect to be continuously available. The FCC order says the Verizon Wireless network outage affected wireless VoLTE 911 traffic in Alabama, Florida, Georgia, North Carolina, South Carolina and Tennessee. VoLTE, short for Voice over Long Term Evolution, carries voice calls over a 4G LTE network instead of an older dedicated voice system.

The distinction matters because a mobile phone can still show signs of network life while a particular call path is impaired. A device may display signal bars, exchange data or interact with parts of the network without proving that a 911 call can be set up and delivered. The public record here does not provide a full service-by-service account, so it would be wrong to say every Verizon service failed or every customer in the six states lost connectivity. The confirmed scope is wireless VoLTE 911 traffic described by the FCC.

According to the FCC order, the outage lasted one hour and forty-four minutes. During those 104 minutes, hundreds of 911 calls did not complete through Verizon Wireless's network. “Hundreds” is the most precise public figure in the reviewed record. It cannot responsibly be turned into an invented exact count, and repeated attempts cannot automatically be counted as unique callers.

The seriousness comes from the failed function, not from an unverified story about what followed. A caller uses 911 because the situation may be urgent and alternatives may be limited. Yet the public sources do not establish that a particular failed attempt caused injury, death, delayed treatment, property loss or another individual outcome. The evidence supports saying that hundreds of emergency calls failed to complete through the carrier's network. It does not support naming victims or assigning consequences the record does not show.

The public event account in this source set comes from the FCC enforcement record. The two published copies of the order are versions of the same document, and the FCC announcement is from the same agency. Federal rule text explains the governing transmission duties, while Verizon's general emergency-information page explains some E911 features. Neither supplies an independent postmortem of the December outage. That source boundary makes careful attribution essential.

The incident also did not begin as a blank-slate technical surprise. The FCC record says the December outage manifested similarly to a Verizon Wireless outage in October 2022. It says Verizon had already performed a root-cause analysis and introduced audits and technical updates intended to prevent recurrence. The later reappearance of a known flawed file is what turns the case from an outage chronology into a test of whether lessons were converted into live operating barriers.

Why a mobile 911 call depends on the carrier

For the caller, 911 is a three-digit number. Behind it is a chain of technical and organisational handoffs. The phone must request the call, the radio network must accept it, the mobile core must process it, and the carrier must transmit it toward the appropriate public safety answering point. A public safety answering point, or PSAP, is the emergency call centre that receives 911 calls and routes them to police, fire or medical services as appropriate.

Federal rules reflect that dependency. Section 9.4 of Title 47 of the Code of Federal Regulations requires covered telecommunications carriers to transmit 911 calls to a PSAP, a designated statewide default answering point or an appropriate local emergency authority. Section 9.10 includes a basic 911 transmission obligation for covered commercial mobile radio service providers. The rule text defines the obligation; the FCC order applies the enforcement record to this specific settlement.

Transmission involves more than recognising the digits. On a modern mobile network, the call must traverse systems that manage subscriber access, voice sessions, security policy and routing. Verizon's current consumer information describes Enhanced 911, or E911, as capable of providing a callback number and approximate location to emergency call takers, subject to limitations. That page offers general context only. It does not describe the 2022 outage, confirm its cause or report its recovery.

The network path is also why configuration management belongs in an emergency-call story. A security-policy update file is a file used to apply network security settings. The public order does not name the exact device, vendor, file format or policy syntax involved here. Even without those details, the operating relationship is clear: a configuration artefact was applied in a network carrying critical call traffic, and the FCC says the resulting outage prevented 911 calls from completing.

Change control is the process for approving, testing, recording and safely applying a network change. Good change control does not mean a network never changes. Mobile networks must be patched, expanded and adjusted constantly. It means the organisation can identify the intended version, distinguish it from an obsolete or unsafe version, test the change under representative conditions, require the right review, observe the result and reverse or contain the change when necessary.

Emergency-call continuity makes those controls especially important because the service must work at the moment a user needs it. A monthly average can hide a short but consequential break. A general network-health indicator can remain green while a particular call path fails. An approval record can show that a person followed a process without proving that the artefact selected was safe. For critical traffic, evidence must connect the approved change to the version that actually reached the live path.

This is also why a regulator's order and an operator's running network play different roles. The FCC can state obligations, investigate an event, preserve an accountability record and impose settlement terms. It does not operate Verizon's live call path. A compliance document can require controls, but only the carrier's systems and operating practices can make a flawed file unavailable, block its reuse and demonstrate that a 911 call still completes after a significant change.

The October warning and the December recurrence

The FCC order provides only a limited public account of the October 2022 outage. It says the December outage manifested similarly to that earlier event. After October, Verizon Wireless performed a root-cause analysis and implemented audits and technical updates intended to prevent the recurrence of configuration and one-way-audio issues. The reviewed record does not publish the October duration, geographic scope, failed-call count or full technical postmortem.

Those limits prevent a claim that the October and December events were technically identical. The decree specifically notes that the December outage did not include the one-way-audio issue associated with the earlier description. Similar manifestation can point to a recurring operational pattern without proving that every mechanism, symptom and affected component was the same.

The most useful way to follow the recurrence is to track one artefact through four states. First, the security-policy update file was known to be flawed after the October experience described by the FCC. Second, it remained in the inventory of policies available for selection. Third, it was selected and reapplied in December. Fourth, the later consent decree imposed explicit obligations concerning unique naming, removal of flawed files and protection against reapplication.

Each state answers a different accountability question. Identification asks whether the organisation knew the artefact was unsuitable. Inventory asks whether that knowledge changed what the live process made available. Selection asks whether people and systems could distinguish the current approved file from a superseded or flawed one. Later obligations show how the settlement translated the recurrence into specific control requirements, without proving how those controls ultimately performed.

The gap between the first and second states is particularly important. Finding a bad file is an analytical result. Removing or quarantining it is an operational change. If the artefact remains selectable, the production environment still contains a route back to the known failure. A report may say the lesson was learned while the live inventory preserves the opposite possibility.

The gap between the second and third states concerns selection and oversight. The FCC record says naming conventions were insufficient, then-current implementation protocols were not followed and required additional oversight was not applied. These facts do not identify a particular software repository, permission system or approval chain. They do show that several expected barriers did not prevent reuse of the file.

December therefore tested the October response in a way a meeting or audit checklist could not. The corrective actions were described as intended to prevent recurrence, yet the known flawed artefact returned to the operating path. The fair conclusion is not that every October measure was useless. The public record does not establish the scope or deployment state of each measure. The narrower conclusion is that the available controls did not prevent this recurrence.

That distinction matters for any operator reviewing an incident. A root-cause analysis can be technically accurate and still fail to produce an effective operational barrier. An audit can identify a problem but leave the unsafe state intact. A technical update can address one symptom while another route to failure remains. The success measure is not whether an action item was marked complete. It is whether the relevant failure is made unavailable, detectable, containable or safely recoverable in the running system.

Trigger, contributing conditions and the root-cause boundary

The consent decree identifies a clear immediate trigger: a Verizon Wireless employee reapplied a known flawed security-policy update file. This is an important fact, but it is not a complete root-cause explanation. Treating it as the endpoint would reduce a multi-layer control failure to the last visible human action.

The FCC record also identifies contributing conditions. The flawed file remained in the available inventory. Naming conventions did not sufficiently distinguish versions. Then-current implementation procedures were not followed. Additional oversight required by those procedures was not applied before the update. Together, those conditions explain why the employee action could reach the live network, even though the public record does not disclose the exact internal sequence.

The distinction between trigger and contributing condition is practical. A trigger is the event that immediately precedes the failure, such as applying a file. A contributing condition is something that makes that event possible or more damaging, such as leaving the file available or failing to require review. A root cause is a deeper explanation that accounts for why the system allowed those conditions to persist. The public materials do not provide a complete independently testable root-cause record.

The record therefore supports a labelled inference, not an invented finding. It suggests that file lifecycle and change control were central to the recurrence because a known flawed artefact remained selectable and the expected procedural barriers did not stop it. It does not establish who designed the inventory, which access controls existed, whether a tool displayed the wrong version, or why a required oversight step was missed.

Nor does the record establish the employee's identity, motive, competence or disciplinary outcome. There is no basis for saying the reapplication was intentional or reckless. A critical control should be designed on the assumption that ordinary people can make mistakes, especially when multiple files look alike or obsolete items remain available. The accountability question is whether one selectable known-bad artefact and one missed oversight step could still reach a critical network path.

Detection, response and recovery remain separate unknowns. The order does not publish the alarm sequence, the time at which Verizon first detected the problem, the escalation path, the rollback mechanics or the steps that restored service. The 104-minute duration is a confirmed recovery metric in the broad sense that it bounds the outage. It does not reveal what happened during those minutes or which action ended the failure.

Keeping those layers separate prevents false precision. The immediate trigger is attributed to the consent decree. The contributing conditions are also described there. The broader file-lifecycle interpretation is analysis based on those facts. Detection and response details are unknown. Recovery duration is known, but the recovery mechanism is not. That structure is more useful than a dramatic narrative because it shows exactly where evidence exists and where it stops.

It also changes how recurrence prevention should be evaluated. Re-training one person might address a particular action, but it would not by itself remove an unsafe file from inventory. Renaming files might improve recognition, but it would not prove a prohibited change cannot be applied. Requiring approval might add oversight, but it would not test how the network behaves under load. The control set needs to address the full path from artefact creation to live effect.

Why a corrective action is not yet a tested safeguard

A corrective action is a response chosen after a problem. A safeguard is a barrier that changes what can happen in the operating system. The two can overlap, but they should not be treated as synonyms. Writing a new procedure is a corrective action. Making a known bad file impossible to select is a safeguard. Scheduling a test is a corrective action. Producing evidence that the test reproduced the relevant conditions and blocked the failure is proof about a safeguard.

This difference explains the central tension in the Verizon record. The FCC says Verizon acted after the October outage. It conducted analysis and introduced audits and technical updates intended to prevent recurrence. Yet the December event occurred after a known flawed file remained available and was reapplied. The intent of the October response and the performance of the December operating path are different forms of evidence.

For a non-specialist reader, a simple analogy is a recalled key. An organisation may discover that a key opens a door it should not, document the problem and tell staff not to use it. Those are meaningful actions. If the key remains on the shared hook with an unclear label, however, the physical system still permits the known failure. A stronger safeguard removes or disables the key, records why, and tests that the door cannot be opened through the same route.

Network configuration is more complex than a key, but the control principle is similar. A versioned inventory should show which file is current, which has been superseded, who changed its state and whether an unsafe file is quarantined. Unique naming should make versions distinguishable. Technical controls should block reapplication of a file deemed unsuitable. Procedures should require review, and the system should preserve evidence that the review covered the exact artefact deployed.

Testing adds another layer. A file may be syntactically valid and still produce an unsafe result in the target network. A significant change should therefore be exercised in a laboratory or other environment that represents the intended network and load. The aim is not to predict every possible outage. It is to test the critical user journey, including a 911 call, under conditions close enough to reveal foreseeable interaction and capacity problems before field application.

The safeguard also needs negative tests. Can a user select a superseded file? Can an automation account apply it? What happens if required certification is missing? Does the deployment stop, warn or continue? Can the same unsuitable protocol be packaged under a new name? Does the inventory retain a reliable record of the decision? A control that passes only the expected happy path has not tested the recurrence route.

Operating evidence matters after deployment too. A change record should connect the reviewed artefact, its hash or stable identifier, the target environment, the approving parties, the test result, the deployment time and the observed outcome. Monitoring should show whether emergency calls continued to complete. Exception reports should record blocked attempts to use superseded or flawed artefacts. These records reveal whether the safeguard is active, not simply documented.

None of these general control examples is a claim about Verizon's undisclosed architecture. The FCC order does not publish the company's complete tooling or later test results. They are ways to understand what evidence would answer the problem exposed by the public facts. The case is valuable precisely because it demonstrates the limit of intention: an action meant to prevent recurrence cannot be credited as a proven barrier when the relevant recurrence still reaches the live path.

What the FCC settlement establishes

The enforcement timeline is separate from the outage timeline. The December event occurred in 2022. The FCC sent an initial letter of inquiry in April 2023. The order records Verizon responses in June, September and October 2023, along with a follow-up FCC inquiry in September. The underlying letters and full company responses are described as being on file but are not reproduced in the reviewed public source set.

On 25 June 2024, the FCC Enforcement Bureau adopted Order DA 24-578, incorporated a consent decree and terminated the investigation under the settlement terms. The order names Cellco Partnership, doing business as Verizon Wireless, as the respondent. A consent decree is a negotiated settlement adopted by the bureau. It is not a court judgment reached after a trial. The one-page FCC announcement summarises the result but says the full order constitutes the official action.

The legal posture requires equally careful treatment of the facts. For purposes of the consent decree and FCC civil enforcement, Verizon Wireless admitted that paragraph 4 accurately describes the facts underlying the investigation. That express scope matters. The article can rely on the paragraph 4 facts with the stated attribution and admission boundary. It should not expand the admission into a general confession outside the decree or describe the settlement as a litigated finding by a judge or jury.

The settlement required Verizon Wireless to pay $1,050,000 and implement a compliance plan. A civil penalty is a monetary agency-enforcement payment, not a criminal sentence. The record does not say that the payment was damages distributed to callers, and it does not establish a compensation amount for individual harm.

The final public action therefore establishes several things: the bureau's adoption of the settlement, termination of the investigation, the limited factual admission, the civil penalty and the agreed compliance obligations. It also preserves the FCC's description of the incident and its connection to the federal 911 rules. Those are stronger and more specific than a mere allegation.

At the same time, the settlement does not establish everything a technical reader might want to know. It does not reproduce the complete internal evidence, decide the case through a court trial or publish an exact failed-call total. It does not identify an injured person. It does not prove that each compliance measure was later implemented successfully. Finality attaches to the adopted order and settlement terms, not to the future effectiveness of remediation.

Four legal states should not be blurred. An allegation is an unproved claim, so that label should not replace the decree's expressly limited admission of the paragraph 4 facts. “Apparent” belongs to the investigation-stage regulatory question and is not a court finding. A proposed remedy is not yet binding, but the compliance-plan terms became final settlement obligations when the bureau adopted the decree. That finality does not extend to later implementation or effectiveness, which the reviewed public record does not prove.

What the compliance plan tries to change

The compliance plan is notable because it translates a short outage description into controls at several stages of the change lifecycle. It requires a senior compliance officer with knowledge of the 911 rules, operating procedures, a compliance checklist, training, reporting and periodic compliance reports. Those governance elements create named responsibility and a record of how the programme is supposed to operate.

At the file level, the decree calls for unique naming that records when a security-policy update was created and superseded. This addresses the selection problem. A filename should not force an operator to guess which version is current. Version status needs to be legible and durable, so a superseded artefact cannot masquerade as the approved one.

The plan also requires removal of flawed files from the available inventory within 24 hours of discovery. That obligation addresses availability rather than recognition. An unsafe artefact that has been labelled accurately but remains selectable still presents a recurrence path. Removing it from the normal inventory changes what the operating process can do.

Another requirement concerns safeguards against reapplying a network change or software protocol deemed unsuitable. This moves beyond a reminder to a prevention control. The useful evidence would show how the system recognises the unsuitable artefact, what identities and automated paths are covered, and what happens when someone attempts to apply it again.

The consent decree also includes employee and vendor certification that business-as-usual procedures were followed for security-policy updates. Certification can create accountability at the handoff between a requester, reviewer and implementer. Its strength depends on whether it is tied to the exact artefact and supported by system records. A generic attestation detached from the deployed version would be easy to complete and difficult to audit.

For significant network changes, the plan calls for testing in a laboratory or other environment that simulates the target network and load before first field application. This addresses a different risk. Correct file identity does not guarantee safe network behaviour. Representative testing asks whether the change performs as expected when it encounters the architecture, traffic and critical call journeys it will face in operation.

The plan further includes 911 risk assessments and reviews of control effectiveness. A risk assessment should identify how a proposed change could affect emergency-call transmission, not merely classify the change by size. A control-effectiveness review should ask whether the barrier prevented or detected the prohibited condition in practice. Those questions connect compliance to the running service.

Taken together, naming, removal, reapplication protection, certification and testing are complementary. Naming makes state visible. Removal reduces exposure. Technical blocking constrains action. Certification records responsibility. Testing examines effect. No single control substitutes for all the others, and a stack of documents cannot substitute for evidence from the live operating path.

The wording “tries to change” is deliberate. These provisions are binding settlement obligations, not speculative recommendations. But the reviewed record does not independently demonstrate their later implementation or effectiveness. The plan tells the public what Verizon agreed to establish. A separate body of current operating evidence would be needed to say whether each safeguard worked over time.

What the public record does not show

The public materials do not disclose the exact network element, vendor, security-policy syntax, file format or change ticket associated with the flawed update. They do not identify the employee or reveal a complete approval chain. Any story that names a firewall model, management platform or internal team would be adding facts not present in the record.

The October event remains only partly described. Its exact date, duration, geography, call impact and full technical cause are not published in the reviewed sources. The statement that December manifested similarly and followed earlier corrective work does not permit those missing details to be filled through analogy.

The record does not publish detection time, alarms, escalation, incident command, rollback or restoration steps for December. It establishes the 104-minute outage duration but not how Verizon discovered the problem or which action restored service. That means detection and recovery quality cannot be rated from the available material.

The impact boundary is also firm. “Hundreds of 911 calls” is not an exact count of calls, callers or emergencies. One person may have attempted more than once; the public record does not say. It does not establish injury, death, delayed response, economic loss or another outcome for an individual. The continuity failure is serious without embellishment.

The underlying Verizon responses cited in the FCC proceeding are not publicly reproduced in the reviewed set. The incident narrative therefore cannot be described as an independent reconstruction from carrier telemetry and multiple unrelated accounts. It is an analysis of the official FCC enforcement record, the codified rules and limited general operator context.

Finally, the public sources do not show whether the later compliance plan was fully implemented, remained in effect beyond its stated term or prevented another comparable incident. A requirement to remove a bad file within 24 hours is evidence of an obligation. It is not evidence of a particular removal event. A requirement to test changes is not a published test result. An effectiveness review is not proof of effectiveness until its method and findings can be examined.

These uncertainties should shape the conclusion, not weaken it. They keep the article focused on what can be tested. The FCC record identifies a recurrence path involving a known flawed artefact, its continued availability and procedural barriers that did not stop reuse. The unanswered question is whether later operating evidence shows that path was actually closed.

What evidence would prove the safeguard works

The strongest evidence would begin with an inventory that treats configuration artefacts as controlled operational objects. Each security-policy file would have a stable identifier, a creation time, a superseded time where applicable, an owner, an approval state and a record of where it may be applied. A known flawed file would be quarantined or removed from the available inventory, while its historical record remained preserved for investigation.

That distinction between removal and erasure matters. An organisation needs to prevent reuse without destroying evidence of what happened. The historical artefact, its hash, the reason it was rejected and the systems it reached can be retained in a restricted evidence store. The ordinary deployment path should not present it as an available option.

Blocked-reapplication tests would provide the next layer. A controlled test could attempt to submit the exact unsuitable artefact, an older superseded version and a renamed copy through each relevant human and automated route. The expected result would be a hard stop, a recorded reason and an alert to the accountable owner. If any route still accepts the change, the recurrence barrier is incomplete.

Change certification should be version-aware. The person or vendor confirming compliance should attest to the stable identifier of the exact file reviewed, the target environment, the required approvals and the test result. The deployment system should verify those fields rather than relying on a free-text statement. After deployment, the observed running version should reconcile with the approved record.

Representative pre-deployment testing would include the critical service journey. For a significant change touching the voice path, the test environment should model the target network and load closely enough to exercise call setup and 911 transmission. Results should identify which conditions were covered, which were not, and what thresholds would block field application. A passing laboratory test should not be described more broadly than its scope.

Production monitoring would then look for the consequence that matters: whether representative emergency calls can complete across the relevant access and core paths. This does not require unsafe live calls to a PSAP. Operators can use authorised test procedures, synthetic checks and coordinated exercises that respect emergency-service operations. The evidence should connect a detected failure to the recent change record and support rapid containment.

Recurrence indicators should be reported over time. Useful measures include attempts to select quarantined files, exceptions to required oversight, deployments whose running hash differs from approval, significant changes without target-load tests, emergency-call probes that fail after a change and time taken to remove a newly identified flawed artefact from available inventory. Zero events can be meaningful only if collection coverage is known.

Independent review can strengthen the conclusion by sampling records and reproducing negative tests. A reviewer should not stop at reading the policy. It should trace an artefact from creation through approval, testing, deployment, supersession and quarantine; attempt prohibited reuse in a safe environment; and verify that the monitoring record matches the observed result.

Evidence of recovery capability belongs in the same programme. The organisation should show that it can identify the last known safe state, reverse or contain a faulty change, preserve call-path visibility during response and document the point at which service is restored. The public order does not reveal how December recovery occurred, so this is a forward-looking evidence test, not a claim about that response.

The standard is not perfection. Complex networks will continue to change, and no control can prove that every future failure is impossible. The practical standard is narrower and more demanding: a known failure route should be made harder to repeat, attempts should be visible, critical effects should be tested, and the organisation should be able to show the resulting evidence without relying on memory or a completed checklist.

That is the difference between a corrective action and a tested safeguard. The first records what an organisation intends to do after an outage. The second changes the available path and leaves verifiable evidence that the known bad file cannot quietly travel from identification back into live service.

Sources