Summary

  • A security certificate in two versions of Ericsson SGSN-MME software expired at 04:30 UK time on 6 December 2018, disrupting multiple operators and countries through a shared mobile-core dependency.
  • Public records support a clear division of practical control: Ericsson controlled the embedded certificate and software lifecycle; operators controlled deployment architecture, incident response, customer communications and service restoration; regulators controlled statutory assessment and future assurance expectations.
  • Ofcom found that O2 had taken appropriate measures and did not breach the applicable UK security duty. That finding did not erase the wider lesson that a provider cannot contract out its legal responsibility or treat a supplier's undisclosed certificate as someone else's continuity problem.
  • Japan's regulator treated SoftBank's outage as a major accident affecting about 30.6 million users and issued a warning and written administrative guidance. SoftBank publicly described rollback, inventory, recovery-procedure and multi-vendor measures.
  • The public repair record is material but incomplete. It does not include Ericsson's final root-cause report, the certificate identity and ownership history, independent validation of the changed software, or evidence that every similarly exposed installation was remediated.
  • Durable accountability requires expiry inventory, named ownership, independently alarmed lifecycle controls, failure-domain diversity, rehearsed recovery, dependency-aware customer notice and evidence that survives management assurances.

The incident was scheduled by a date, but its consequences were architectural

At 04:30 in the United Kingdom on 6 December 2018, an embedded security certificate reached its expiry time. O2's mobile network began losing service. In Japan, where the same instant was 13:30, SoftBank's network failed minutes later. Ericsson eventually said that the disturbances concerned certain core-network nodes used by a limited number of customers in multiple countries, that two particular versions of its Serving GPRS Support Node and Mobility Management Entity software were implicated, and that an expired certificate was the main issue identified in its initial root-cause analysis. Its official update distributed through Cision said the faulty software was being decommissioned and that most affected services had been restored during the day.

That description establishes the trigger, but it does not fully explain the accountability problem. Certificates are time-bounded credentials. Under the Internet Engineering Task Force's X.509 certificate and certificate revocation profile, RFC 5280, a certificate carries a validity interval defined by notBefore and notAfter; a relying system must be able to determine whether the certificate is valid at the relevant time. The precise role of Ericsson's certificate has not been publicly documented in enough detail to reconstruct its validation path. What is confirmed is that its expiry interacted with the software in a way that stopped core-network functions. The expiry instant was deterministic. The nationwide and cross-border consequences were products of where the software sat, how uniformly it had been deployed, and what recovery choices were available after the failure.

The affected SGSN-MME was not an optional customer-facing application at the network edge. Ericsson's current product description of the SGSN-MME describes a packet-core control function supporting mobility and session management across GSM, WCDMA and LTE access. That current description cannot by itself establish every detail of an operator's 2018 design. It does explain why failure in this class of component can have effects far beyond one server: the function helps devices attach, remain reachable and establish data sessions. A common software defect replicated across the nodes intended to provide capacity or geographic resilience can therefore defeat physical redundancy at the same instant.

This distinction matters. A certificate expiry is sometimes framed as a housekeeping omission: renew the credential, restart the service, move on. In a national mobile core, however, the relevant unit of control is not the certificate alone. It is the combined system of software release governance, credential inventory, alarm independence, operator acceptance, deployment diversity, rollback readiness, subscriber reconnection and communications. The December 2018 event tested that entire system. Its value as an accountability case lies in the way a small, scheduled condition exposed a large shared failure domain.

An auditable timeline shows two national incidents with one vendor trigger

The United Kingdom record is unusually detailed because Ofcom investigated. Its decision concluding the O2 network-outage investigation states that the outage began at 04:30 and lasted almost 23 hours. It affected all of O2's approximately 25 million direct customers, as well as connections supplied through mobile virtual network operators. At different points, mobile data, calls and messages were unavailable. O2 convened its major incident team at 04:50, 20 minutes after the first failure, and established communications bridges with Ericsson. A board-level major incident team chaired by O2's chief executive oversaw the response.

The technical path to recovery was neither a single certificate replacement nor an immediate restart. Ofcom recorded that it took about 12 hours to arrive at a successful fix. O2 and Ericsson then restored service in phases because reconnecting a national subscriber population all at once risked congestion and overload. O2 restored 2G and 3G service at about 21:30. It began restoring 4G at about 23:30 and completed that work at 03:12 on 7 December. The phased sequence is important evidence: even after engineers neutralized the initiating software condition, the network still had to manage a potentially destabilizing recovery wave.

Ofcom's public case record adds the procedural timeline. The regulator opened its investigation on 22 February 2019 and closed it on 1 November 2019. It assessed whether O2 had taken appropriate measures to maintain network security and availability and whether the provider had taken all appropriate steps to restore service. The inquiry ended without a breach finding, but with detailed expectations for how providers should assure supplier certificate controls in the future.

Ofcom also incorporated the outage into its Connected Nations 2018 report. The report noted the effects on O2's own customers and on wholesale brands including Tesco Mobile, Sky Mobile, TalkTalk Mobile, giffgaff and Lycamobile. It observed that voice and messaging were also affected during parts of the incident and recovery. It identified practical resilience lessons, including checking key network elements, following good practice when reconnecting large numbers of users, and considering greater control-plane redundancy for networks supporting wholesale customers.

O2's parent company supplied the commercial-response detail. Telefónica's 2018 Integrated Management Report recorded the event as a significant outage caused by expiry of an Ericsson certificate and described 2G and 3G disruption lasting about 17 hours and 4G disruption lasting about 24 hours. O2 offered direct monthly customers a credit equivalent to two days of monthly charges, pay-as-you-go customers an additional 10 percent on a January top-up, and mobile-broadband customers a 10 percent discount on a January purchase. The report therefore confirms a concrete transfer of cost from users to the operator, although it does not publish a total monetary value for the credits or the full incident cost.

SoftBank's public timeline begins at 13:39 Japan Standard Time. Its 6 December incident release says nationwide LTE service for SoftBank and Y!mobile customers became unavailable or difficult to use until 18:04, a period of four hours and 25 minutes. The effects also reached Ouchi-no-Denwa fixed-line service and part of SoftBank Air. Traffic moved toward 3G, causing congestion there. SoftBank said all Ericsson packet switches in Tokyo and Osaka were affected, that the software had been in use for nine months, and that Ericsson had reported simultaneous incidents at operators in 11 countries. SoftBank restored service by applying older software to all affected switches.

SoftBank's 12 December incident explanation sharpened the control evidence. It attributed the failure to incorrect handling of an expiry in Ericsson switch software and said the relevant expiry information had been embedded by Ericsson at shipment and could not be inspected by SoftBank. The company said it had moved to software that handled the expiry correctly and had checked major communications equipment, including the affected systems, for similar issues. That is a direct operator statement, not an independent technical examination, but it is central to locating who could see and alter the initiating condition before 6 December.

At a 19 December management briefing, summarized in SoftBank's official corporate news account, the operator put the impact at approximately 30.6 million lines. The account provides a minute-by-minute recovery sequence: the fault began at 13:39; the first website notice appeared at 14:19; separation work started at 14:45; LTE traffic controls were applied at 14:57; the likely LTE switch problem was identified at 15:54; updating all LTE switches began at 16:22; western Japan recovered at 17:35; and national restoration was declared at 18:04. This record does not make the two national recoveries directly comparable. Their network designs, subscriber-state conditions and reporting conventions were different and are not fully public. It does show that operators facing a shared trigger used different technical recovery paths and restored service on materially different schedules.

Practical control was divided, but it was not evenly distributed

Accountability becomes clearer when it is separated into control before failure, control during failure and the duty to prove repair. Ericsson had direct control over the software source, release packaging and the certificate condition embedded in the affected versions. The public record indicates that O2 was not told about the hardcoded certificate or its risk and that SoftBank could not inspect the embedded expiry information.

Those facts placed the most direct preventive controls with the vendor: maintain an authoritative inventory, assign an owner, renew or remove the credential, test behavior on and beyond the expiry boundary, alert independently of the affected code path, and notify every exposed customer with enough lead time to act.

Operators nevertheless retained substantial control. They selected and deployed releases, negotiated supplier obligations, designed topology and failure domains, approved maintenance and rollback procedures, monitored service health, declared incidents, communicated with customers and regulators, and managed the mass reconnection of devices. Some preventive controls may have been impracticable for a credential invisible inside proprietary software. Architectural containment, supplier assurance, recovery rehearsal and public notice remained operator responsibilities.

The fact that an operator could not inspect one hidden field does not mean it had no influence over the surrounding continuity system.

Ofcom made that legal and operational boundary explicit. Its decision said a communications provider cannot contract out its statutory obligations by outsourcing network functions to a third party. At the same time, the regulator examined what O2 could reasonably have done in the circumstances.

It found that O2 had robust contractual, testing and risk-management arrangements; that Ericsson had not disclosed the hardcoding or the associated risk; that the specific error was not designed for in O2's acceptance testing; and that adding a control to detect this particular issue in advance was not technically or commercially feasible for O2 on the evidence available. Accountability, in this decision, was not strict liability for every supplier defect. It was an evidence-based test of whether the licensed provider had taken measures proportionate to the risks it could know and control.

The statutory baseline in force at the time can be read in the historical version of section 105A of the Communications Act 2003. It required public electronic communications network and service providers to take measures appropriately to manage risks to security, including measures aimed at minimizing the impact of security incidents on end users and interconnected networks. Ofcom's no-breach finding must be understood against that standard and the investigation evidence. It was not a declaration that the outage was harmless, that certificates did not require governance, or that operators could ignore common-mode supplier risk after the lesson became known.

Mobile virtual network operators and enterprise customers occupied a weaker control position. They depended on O2's underlying network and could not patch Ericsson's core software or sequence national subscriber recovery. Their controls were contractual visibility, business-continuity planning, multi-network options where justified, customer communications and escalation. The Connected Nations record demonstrates why the supplier chain matters: one core-network condition reached brands that many consumers would not necessarily associate with O2.

A dependency map that stops at the retail brand would not have described the actual failure domain.

Regulators held a different kind of control. They could require incident reports, compel evidence, assess statutory compliance and convert a one-off event into forward-looking assurance expectations. They could not renew the embedded certificate or restore a switch. Their accountability role was to test whether those with operational control had behaved appropriately and whether the sector changed its controls after the event. Investors and boards, meanwhile, controlled incentives, budgets and oversight.

They needed to ask whether high availability claims covered common software dates, whether remediation was verified across the installed base, and whether supplier concentration was visible as a continuity exposure rather than only a procurement fact.

The users affected by the outage had almost no preventive control. They could retry, move to Wi-Fi, use another provider if they already had one, or wait. That asymmetry is why public accountability cannot rest on a claim that customers were later informed. Notice and compensation matter, but they operate after risk has already been allocated to people who could neither inspect the certificate nor choose how the packet core was built.

Harm extended beyond lost data sessions, while the full cost remains unknown

The confirmed scale is large but must be stated carefully. O2 reported approximately 25 million direct customers, plus wholesale connections. SoftBank later described approximately 30.6 million affected lines. Those figures use different units and may include multiple subscriptions held by one person, so they should not be added and presented as a count of unique individuals. Ericsson and SoftBank described effects in multiple countries, but the public primary-reference reviewed here does not support a complete operator-by-operator global total.

Service effects were broader than slow mobile internet. Ofcom found that data, calls and messages were unavailable at different times in the UK. SoftBank reported nationwide LTE failure, congestion on 3G, disruption to a fixed-line product and partial impact on a wireless broadband product. Japan's Ministry of Internal Affairs and Communications treated the event as a major accident under telecommunications law. Its notice concerning receipt of SoftBank's serious-accident report documents the formal reporting step after the outage, anchoring the event in a legal process rather than only in corporate incident communications.

The ministry's later 23 January 2019 warning and administrative-guidance announcement recorded an approximately four-hour-and-25-minute outage affecting about 30.6 million users. It emphasized the mobile network's role in carrying emergency calls and serving as a social lifeline, and described the social impact as extremely large. That statement supports a finding of systemic risk. It does not establish that a particular emergency call failed, that a specific person was injured, or that the outage caused a death. Those outcomes are not documented in the cited public record and should remain unclaimed.

Financial harm is also only partly visible. O2's credits are confirmed. Customers and businesses also lost time, transactions and access, but no audited total is public. SoftBank described remediation that included equipment review, procedural changes and greater diversity; the public materials do not isolate their costs. Ericsson apologized and worked on restoration, but its update did not disclose compensation paid to operators, engineering expenditure or contractual liability.

Any single global loss figure would therefore mix confirmed credits with unsupported assumptions about customer behavior, service-level agreements and downstream business interruption.

There were less visible accountability costs. Incident teams worked across vendor and operator boundaries for hours. Regulators gathered and reviewed technical and contractual evidence. Operators had to explain a failure they did not fully control to customers whose relationship was with the operator, not the equipment supplier. Wholesale brands inherited a communications problem from a network layer outside their direct management. These are real categories of burden, yet the public record does not provide enough data to monetize them responsibly.

The most durable harm was the discovery that supposed redundancy could share a hidden expiry. If identical software and credential state are reproduced across nodes, adding more nodes can add capacity without creating independence from that state. This is a supported inference from SoftBank's statement that all relevant Ericsson packet switches failed and from O2's national effects. It is not proof that every node in every affected network had an identical topology, nor does it establish the exact software execution path. The distinction protects analysis from turning an architectural lesson into an invented forensic report.

The regulatory record cleared O2 of breach but raised the assurance standard

Ofcom's decision is more useful than a simple guilty-or-not-guilty summary. The regulator concluded that O2 had taken appropriate measures to manage security risks and had taken all appropriate steps to restore service. It therefore found no breach of section 105A(4). Its reasoning credited O2's contractual framework with Ericsson, testing arrangements, risk management, rapid incident organization, senior oversight, collaboration with the vendor and cautious phased restoration.

It also accepted that the undisclosed hardcoded certificate and this specific failure mode were outside what O2 could reasonably have identified through its existing acceptance process.

That conclusion established a boundary for retrospective responsibility under the evidence then available. It did not freeze the boundary for future incidents. Once the failure mode was known, providers could no longer treat embedded certificate expiry as wholly unforeseeable. Ofcom stated future expectations that providers seek assurance about suppliers' certificate policies, lifecycle monitoring and exception handling; ensure that departures from policy are documented and reviewed; test certificate use carefully where availability may be affected; and maintain proportionate processes for certificates across their networks.

Ofcom also said its own security assurance work would specifically ask about certificate use and expiry.

This is the core accountability shift. Before 6 December, Ofcom accepted evidence that O2 could not reasonably have added a specific preventive control. After 6 December, the regulator converted the event into sector knowledge. A provider assessing a comparable hidden credential in later years would face a different foreseeability question. Durable accountability therefore depends not only on whether conduct met the standard at the instant of failure, but also on whether organizations absorbed a documented lesson into procurement, acceptance, operations and audit.

The Japanese record took a different formal route. The Ministry of Internal Affairs and Communications received SoftBank's major-accident report in late December and issued a strict warning and written administrative guidance in January. The formal guidance letter is the authoritative written instrument associated with the ministry's announcement. The ministry called for concrete recurrence-prevention measures and reporting, while its public account highlighted internal and external coordination, information provided to users and industry-wide sharing of lessons. The action was directed at SoftBank as the licensed operator, even though the initiating software condition was attributed to Ericsson. That allocation mirrors the UK principle in practical terms: the operator remains answerable to the sector regulator for continuity and public communications, while the vendor's control over the defect remains part of the factual analysis.

The two regulatory outcomes should not be collapsed into a contradiction. Ofcom conducted a detailed statutory investigation and found O2 compliant. The Japanese ministry issued a warning and administrative guidance after SoftBank's report. The legal frameworks, procedures and evidentiary records differed. Neither outcome by itself establishes civil liability between vendor and operator. Neither public record publishes the relevant contracts, warranty provisions or any confidential settlement.

Legal accountability in the public domain is therefore strongest at the level of operator duties and regulator findings, not private allocation of damages.

The case also demonstrates why regulatory review must examine suppliers without pretending that the regulator directly operates them. Ofcom gathered substantial information from Ericsson and used it in evaluating O2. It could compare the operator's controls with what the vendor had disclosed. A serious review of outsourced critical infrastructure needs that reach. If the inquiry considered only documents already visible to the operator, the party with the most direct knowledge of an embedded condition could remain outside the evidentiary picture.

Repair evidence exists, but assurance claims still have gaps

Ericsson's same-day update is the first layer of repair evidence. It said the faulty software was being decommissioned, that services were being restored, and that an initial analysis had identified an expired certificate while a full root-cause analysis continued. The statement included an apology from the chief executive. It did not publish the final analysis, identify the certificate, explain how it entered the releases, state how long it had been known, or list all affected customers and versions. A same-day operational update is appropriate for restoration, but it is not a substitute for a durable technical closure record.

O2's repair evidence is stronger on response process. Ofcom reviewed incident governance, technical collaboration, restoration sequencing, supplier management and post-incident changes. The regulator found the response appropriate and described future certificate-assurance measures. Telefónica documented customer compensation. The limitations are still material: the public decision summarizes evidence rather than publishing configuration artifacts, test results or a full release-by-release remediation inventory. The absence of those artifacts in public does not mean they never existed.

It means external readers cannot use them to verify complete removal of the risk.

SoftBank disclosed a broader set of technical and organizational measures. Its December briefing said it would complete an inventory of certificates in commercial equipment, shorten the procedure for emergency startup of older software, change designs so a similar condition would not stop the system, add LTE switches and promote multi-vendor deployment. It also said it had checked major communications equipment for similar problems. These measures map sensibly to prevention, recovery and containment.

They are company claims, however, and the cited public record does not include independent test reports showing that the inventory was complete or that a later expiry simulation succeeded.

The difference between action and proof is central. “We checked certificates” is an action statement. Durable proof would identify the population checked, discovery method, accountable owners, exceptions, expiry horizons, alert paths, closure dates and independent sampling. “We added redundancy” is an action statement. Durable proof would show that redundant paths do not share the same credential, software date or management dependency. “We can roll back” is an action statement.

Durable proof would show a rehearsed rollback under load, known subscriber-state effects, restoration time objectives and criteria for choosing rollback over an in-place fix.

There was already a relevant control model before the incident. In November 2017, the US National Institute of Standards and Technology published a TLS server certificate management project description. It described the outage risks created by expired certificates and called for a formal inventory, identified owners, automated discovery and monitoring, status reporting and organized remediation. That publication was not written for Ericsson's mobile-core implementation, and the public record does not establish that the embedded credential was a conventional TLS server certificate managed by the same tooling. Its evidentiary value is narrower: the general governance problem of certificate expiry was documented before December 2018, even if this particular software behavior and deployment were not known to O2.

NIST later published the more complete SP 1800-16 practice guide on securing web transactions through TLS server certificate management. Again, a web-certificate reference architecture cannot be transplanted mechanically into a carrier packet core. The enduring principles are transferable: discover certificates, bind them to accountable owners and systems, monitor lifecycle state, protect management access, automate safe renewal where suitable, test changes and retain operational evidence. For embedded or proprietary credentials, the implementation may need vendor attestations, software bills of materials for credential dependencies, lab time manipulation and contractually required advance notice instead of ordinary enterprise discovery tools.

Current telecom security specifications now make the availability concern explicit. The January 2026 edition of ETSI TS 133 310 on network-domain security and public key infrastructure includes certificate lifecycle provisions and expiry-related alarm expectations intended to mitigate service unavailability. It is a current benchmark, not evidence of the exact requirements applied to Ericsson or the operators in 2018, and it does not prove remediation in any installed network. It demonstrates what a durable control environment should now be able to operationalize: expiry must produce managed action before it produces loss of service.

Counterfactuals separate preventable failure from unknowable detail

A disciplined counterfactual asks what specific control would have changed the result without assuming facts that remain private. The strongest prevention counterfactual is an authoritative vendor-side credential inventory linked to release and customer deployment. If the embedded certificate had a named owner, monitored expiry horizon and escalation independent of the affected software, the date could have triggered renewal, replacement, removal or a customer upgrade campaign before 6 December. This is a supported inference grounded in the deterministic validity interval and established certificate-management practice.

It does not reveal why Ericsson's actual controls failed.

An operator-side counterfactual is less direct but still meaningful. A contract and release-assurance process could require the supplier to disclose every credential capable of affecting availability, its expiry date, renewal design, alarm behavior and affected versions. Operators could require a machine-readable expiry inventory or a signed attestation for proprietary components they cannot inspect. Ofcom found that O2's existing controls were reasonable for the undisclosed risk, so this should not be rewritten as a control O2 was legally required to have in 2018. It is a forward-looking control made reasonable by the incident itself.

Containment provides a second counterfactual. SoftBank said all affected Ericsson packet switches in Tokyo and Osaka failed. If redundant nodes shared the same software and expiry condition, their physical separation did not create logical independence. Version diversity, staged deployment, credential diversity or a separately implemented standby could have reduced the common failure domain. Multi-vendor architecture is one possible route, and SoftBank named it as a permanent measure. It is not costless: multiple vendors can create interoperability, operational and assurance complexity.

The accountability test is not whether an operator purchased a second brand, but whether it identified which failures remained common across supposedly independent paths and justified the residual risk.

Recovery offers the clearest observed comparison. SoftBank rolled back to older software and restored LTE in four hours and 25 minutes. O2 and Ericsson spent about 12 hours reaching a successful fix, then restored generations in phases and completed 4G recovery nearly 23 hours after onset. A prevalidated rollback package, current configuration backup, practiced decision authority and subscriber reconnection plan could shorten a future outage. It would be unsound to infer from duration alone that one operator's team was better.

The public record does not disclose equivalent topology, load, safety constraints or the defects present in each rollback option.

Communications form a third counterfactual. Earlier notices, clearer statements about affected services and coordinated messages across wholesale brands would not repair the network. They could reduce wasted user effort, help organizations invoke continuity plans and allow emergency-service users to seek alternatives. Japan's regulator specifically treated user information and coordination as part of the remediation problem. The durable measure is not the volume of updates but whether they are timely, consistent, accessible through unaffected channels and explicit about what remains unavailable.

Finally, compensation addresses a portion of harm after the fact. O2's credits recognized service failure without requiring every customer to prove an individual loss. They did not compensate every downstream business interruption, nor did they prevent recurrence. A mature accountability system uses compensation, technical remediation and evidence disclosure as separate tools. None can stand in for the others.

Confirmed facts, supported inferences and unknowns

Confirmed facts. Ericsson said two specific SGSN-MME software versions used by a limited number of customers in multiple countries were involved and identified an expired certificate as the main issue in its initial analysis. Ofcom found that an embedded or hardcoded security certificate expired at 04:30 UK time, causing O2's outage; it documented almost 23 hours of disruption, about 25 million direct O2 customers plus wholesale connections, a 20-minute incident-team activation, a roughly 12-hour path to a successful fix and phased restoration. Ofcom found no breach by O2 and considered its preventive and restorative measures appropriate. SoftBank reported a nationwide four-hour-and-25-minute LTE outage, congestion and related service effects, an older-software rollback, Ericsson switches in Tokyo and Osaka, and information from Ericsson that operators in 11 countries experienced simultaneous incidents. Japan's ministry recorded about 30.6 million affected users and issued a warning and written guidance. Telefónica documented O2 customer credits. SoftBank publicly described inventory, recovery and diversity measures.

Supported inferences. The deterministic expiry should have been detectable by whoever held an accurate inventory and visibility into the embedded credential. Replication of the same vulnerable software and credential state created correlated risk that physical node redundancy alone could not contain. The most direct pre-failure control sat with Ericsson because operators said the relevant condition was undisclosed or uninspectable, while operators retained control of architecture, supplier assurance, response, restoration, notice and customer remedy. A prevalidated rollback and mass-reconnection procedure could reduce recovery time in a comparable event. After the incident, embedded certificate expiry became a foreseeable class of continuity risk for operators and vendors even where the exact code-level failure mechanism remained proprietary.

Unknowns. The public record does not identify the certificate by subject, issuer, serial number, fingerprint or exact validity dates beyond the observed expiry time. It does not publish the certificate's precise protocol role, the code path that converted expiry into node failure, who created or approved it, which review gates it passed, what alerts existed, or why those alerts did not prevent the outage. Ericsson's final root-cause report is not public in the cited record. There is no complete public list of all countries, operators, nodes or software releases affected. The contracts and any private allocation of liability are unavailable. Total losses, vendor payments and remediation costs are not audited publicly. The record does not establish a cyberattack or malicious act. It does not provide independent, fleet-wide verification that every repair and certificate inventory was complete.

These categories should not be blurred. Confirmed facts are strong enough to allocate practical control at a functional level. Supported inferences show how continuity controls could have interrupted the sequence. Unknowns prevent claims about individual fault, motive, negligence inside Ericsson, private damages or the completeness of remediation. Accountability is strengthened, not weakened, when those limits are explicit.

A durable accountability test for shared mobile-core software

The incident deserves closure only when a vendor and each relying operator can answer a set of evidence questions. The first is scope: which products, versions, nodes, networks and customers contain a certificate or other time-bound dependency capable of affecting service? An inventory should include embedded credentials that ordinary network scanners cannot see. It should map each item to a software release, deployment population, business service, owner and expiry horizon. A percentage without a defined denominator is not sufficient.

The second is ownership: who can renew, replace or remove the credential, and who must act when the owner is a supplier? Ownership must be a role with authority, budget and an escalation route, not an email alias. Vendor contracts should specify disclosure, advance-warning periods, emergency fixes, supported rollback versions, evidence delivery and the treatment of end-of-life software. Operator acceptance should verify that those obligations are reflected in actual release data.

The third is detection independence: would the warning survive the same condition that threatens the service? An alarm generated only by the expiring component may disappear when the component stops. Lifecycle monitoring should therefore use a separately managed source of truth and multiple warning thresholds. Exceptions, including hardcoded certificates and credentials that cannot be renewed automatically, should be documented, approved, time-limited and reviewed at an appropriate level.

The fourth is boundary testing. A lab should test behavior before, at and after relevant validity limits, including clock shifts, certificate replacement, restart, failover and rollback. Testing should cover control-plane availability and the effect on devices attempting to attach or re-establish sessions. A generic acceptance test that proves normal operation on today's date does not answer what happens on the expiry date. Evidence should identify the release tested, test clock, certificate state, expected alarms, observed behavior and unresolved deviations.

The fifth is failure-domain independence. Ericsson's later discussion of robust critical networks explains that control-plane nodes such as an MME can serve very large populations and that recovery design must account for signaling load and geographic redundancy. That vendor-authored architecture discussion is not proof of repair after the 2018 event. It is useful for evaluating the right question: do redundant nodes merely sit in different places, or do they avoid the same software, credential, timing, orchestration and management failure? Operators should document common dependencies across regions, generations and wholesale services, then test failure of those common layers.

The sixth is recoverability under load. Teams need a known-good rollback artifact, compatible configuration, protected access, authority to choose rollback, and a plan for staged reconnection. Exercises should measure detection, diagnosis, decision, deployment and service-restoration time separately. They should include the secondary load created when millions of devices retry. A successful lab restart of one node is not proof that a national population can return safely.

The seventh is communications accountability. Operators should maintain a dependency-aware contact map covering retail brands, wholesale customers, enterprise channels, emergency-service stakeholders and regulators. Notices should distinguish confirmed service effects from investigation, state practical alternatives where available and use channels independent of the failed mobile service. Timestamped records should show when the organization knew a material fact and when it communicated it.

The eighth is remedy and learning. Customer credits or other remedies should have transparent eligibility and should not depend on users knowing the hidden technical cause. Post-incident learning should feed software procurement, release approval, business continuity, regulatory assurance and board risk reporting. The lesson should be tested later: sample installed systems, inspect evidence, simulate expiry and verify closure of exceptions. A policy revision with no operational sampling is documentation, not assurance.

The ninth is publicly defensible closure. Critical-infrastructure suppliers cannot publish every sensitive configuration detail, but they can disclose enough to establish scope, cause category, remediation population, validation method and remaining risk. A useful closure statement would say which product families and release ranges were affected, how all customers were identified and notified, what lifecycle control changed, how expiry behavior was tested, who independently reviewed the result and whether any exceptions remain. Sensitive identifiers can be withheld without reducing the account to “software fixed.”

Applied to December 2018, this test produces a mixed result. Restoration is confirmed. O2's response and compliance were examined by an independent regulator, and the regulator found its measures appropriate. SoftBank disclosed concrete immediate and permanent actions, and Japan's ministry required recurrence-prevention reporting. Ericsson identified the initial cause category and said it was decommissioning faulty software. Yet public closure remains incomplete because the final vendor analysis, installed-base scope, control-owner history and independent fleet-wide validation are not available.

That gap does not justify an accusation beyond the evidence. It justifies a durable request for proof. The next expiry should be discoverable years in advance, visible to both the party that embeds it and the operator whose customers depend on it, tested at its boundary, isolated from common-mode failure and recoverable through a rehearsed path. Boards and regulators should be able to inspect those claims without waiting for another synchronized outage.

Accountability after the clocks moved on

The December 2018 outage was not important because certificates are exotic. It was important because an ordinary time limit inside shared proprietary software crossed organizational and national boundaries at once. Ericsson controlled the embedded condition and the affected software. O2 and SoftBank controlled different parts of deployment, resilience, recovery and customer response. Regulators assessed the licensed operators and translated the incident into stronger expectations. Users bore the immediate loss of connectivity despite controlling none of those upstream decisions.

The fairest reading of the record avoids two easy errors. One is to absolve operators because the vendor supplied the defect. The other is to assign every consequence to operators despite evidence that the triggering condition was hidden and, in O2's case, not reasonably detectable under the controls Ofcom examined. Practical accountability follows control, knowledge and legal duty at each stage. It can be shared without becoming vague.

By 2026, the durable question is no longer whether this precise expiry was foreseeable to every operator in 2018. It is whether vendors and carriers can now prove that time-bound dependencies in critical network software are inventoried, owned, monitored, tested, diversified and recoverable. The incident supplied the warning. The accountability test is whether the evidence of change will outlast the memory of the outage.