Google Cloud's Belgium disk loss showed where locality stops

Summary

In August 2015, Google Compute Engine Standard Persistent Disks in the europe-west1-b zone experienced read errors after four successive lightning strikes affected the local utility grid serving a European data centre. Google later reported that a very small fraction of allocated Persistent Disk space in the zone suffered unrecoverable recent writes.
The accountability issue is not whether the percentage was large. It is whether customers understood that a zonal Persistent Disk, even with provider-managed redundancy inside the zone, was still inside a physical failure domain and not a substitute for independent snapshots, regional replication, or application-level backup.
Google controlled physical-site resilience, storage hardware susceptibility, power-event handling, Persistent Disk durability language, status reporting, and the clarity of backup guidance. Customers controlled workload architecture, snapshot schedules, recovery objectives, replication choices, and whether locality requirements were confused with recoverability.
The practical repair record should distinguish restored service, unrecoverable data, available snapshot workaround, hardware and software changes, backup guidance, and customer evidence. In a cloud incident involving data loss, a green status page cannot be the whole recovery proof.

A tiny percentage can still be a hard failure

The Google Cloud Belgium incident is sometimes remembered as a curiosity because the percentage of permanently lost storage was extremely small. That is the wrong first lens. For a customer whose disk contained the unrecoverable recent write, the percentage did not matter. The relevant question was whether the customer had a recoverable independent copy, whether the application could tolerate the recovery point, and whether the provider's durability language had made the remaining physical-site risk clear enough before the event.

Google's public Compute Engine Incident #15056 began on August 13, 2015 for Persistent Disks in europe-west1-b. The status page first reported read errors for customers with machines in that zone, then explained that less than 1 percent of disks in the zone were susceptible to degraded performance, then that less than 0.1 percent were experiencing read failures on some blocks. The incident record also told affected customers that restoring from snapshots was a workaround, while creating new Persistent Disks and restoring from snapshots were not affected.

Media reports that captured the later incident explanation, including Data Center Dynamics' report on lightning and data loss and Silicon UK's account of the outage cause, recorded Google's statement that four successive lightning strikes on the local utility grid caused a brief loss of power to storage systems hosting disk capacity for GCE instances in europe-west1-b. Google said almost all data was committed to stable storage, but in a very few cases recent writes were unrecoverable, resulting in permanent data loss on Persistent Disk. The widely repeated figure was less than 0.000001 percent of allocated Persistent Disk space in the affected zone.

That record supports both restraint and seriousness. It would be wrong to describe the event as widespread data destruction across Google Cloud. The affected service was Standard Persistent Disk in one zone; SSD Persistent Disk, snapshots, and Local SSDs were reported by postmortem indexes and contemporary coverage as outside the permanent-loss population. It would also be wrong to dismiss the incident because the denominator was huge. Data durability is a binary fact for the record that matters. A tiny unrecoverable percentage is still a permanent loss for someone.

The accountability question is therefore not "Why did lightning exist?" Lightning is an external hazard. The question is who controlled the design choices that let a repeated utility-grid power event reach recently written disk state, who controlled the clarity of durability and backup guidance, and who controlled the customer architecture that either did or did not have an independent recovery point. The trigger was physical. The root accountability issue was the boundary between provider-managed local durability and customer-managed recoverability.

Locality and durability are not the same promise

Cloud locality solves real problems. A customer may choose europe-west1 for latency to Belgian or European users, for procurement reasons, for lower carbon characteristics, or for data-location commitments. Google's current cloud locations page and Compute Engine regions and zones documentation explain that resources live in regions and zones, and that zones and regions are logical abstractions of underlying physical resources. That abstraction is useful because customers do not need to manage buildings. It is dangerous if customers infer that a zonal resource has escaped the physical failure domain.

Data sovereignty and locality are about where data is stored or processed. Recoverability is about whether another usable copy exists after failure. A disk can satisfy a location requirement and still be the wrong durability architecture for a database if its only recoverable state lives in the same zone and on the same storage class. A snapshot can satisfy recovery but may have its own location choices. A regional disk can increase availability across zones but may not satisfy every recovery-point target. A second provider can reduce common dependence but may increase operational complexity and data-governance risk.

These are different dimensions.

Google's current data residency terms and European commitments material address where customer data may reside for supported services. They do not turn every local resource into an independent backup. Similarly, the Persistent Disk product page describes durable block storage, and the Compute Engine Persistent Disk documentation states that Persistent Disk has built-in redundancy to protect against equipment failure and maintain data availability through maintenance events. Those are meaningful provider commitments. They are not a guarantee that every possible site-scale hazard will leave zero recent writes unrecoverable in every configuration.

The 2015 incident exposed the interpretive gap. A customer might read "persistent" as meaning the disk outlives a virtual machine, which is correct. Another might read it as meaning the disk is immune to data loss, which is not a safe inference. A customer might read "Europe" or "Belgium" as the main compliance decision and stop there. The incident shows that location is not a recovery plan. The same local placement that helps latency and policy can concentrate physical risk if no independent backup exists.

Provider language should therefore be explicit about failure domains. A zonal Persistent Disk is durable within its design but remains tied to a zone. Snapshots, regional disks, replication, and application backup change the failure model. Customers need that distinction before an incident, not only after a status page tells them to restore from snapshots. The highest-value disclosure is a plain mapping from storage choice to failure domain, recovery point, recovery time, and customer duty.

The physical trigger belongs in the cloud accountability record

The cloud can make physical infrastructure disappear from the customer's daily work, but it does not make physical hazards disappear. Power systems, batteries, storage controllers, firmware, racks, electrical distribution, and utility-grid events remain part of the service. The customer pays the provider to manage those layers because the provider has greater scale and expertise. That makes physical resilience a provider duty, while leaving application recovery architecture partly with the customer.

The incident explanation reported by multiple outlets said automatic auxiliary systems restored power quickly and storage systems were designed with battery backup, but some recently written data was located on systems more susceptible to power failure from extended or repeated battery drain. That sentence matters because it distinguishes a single lightning strike from a repeated physical stress that found a vulnerable subset of storage.

It also shows why the "old disks" framing used in some reporting should be treated carefully: public articles described hardware susceptibility, but the public status record does not publish every component, age, or internal engineering decision.

Google reportedly said it conducted a wide review across electrical distribution, computing hardware, and the software controlling the Persistent Disk layer, and that it was upgrading storage hardware to be less susceptible to this type of power failure. Data Center Knowledge's updated report recorded that Google was replacing storage systems with more power-resilient hardware and that much Persistent Disk storage was already on newer hardware. Those are responsive measures. They should be understood as provider-side controls over the physical and storage stack, not as customer architecture actions.

The provider also controlled incident status. The Cloud Status page supplied repeated updates, impact percentages, and snapshot workaround guidance. That record is materially better than silence. It did, however, move from read errors and degraded performance to permanent loss as investigation progressed. Customers needed to know which disks had read errors, whether snapshots were usable, whether new disks could be created, which writes were unrecoverable, and whether storage was safe for new workloads. In a data-loss event, impact classification is not only about service availability; it is about recoverable state.

The status page ended when Google marked the incident resolved. For customers who restored from snapshots, recovery continued through application validation, data reconciliation, and possible loss of recent transactions. That distinction is essential. Provider service restoration means the storage service is operating. Customer recovery means the workload has a consistent data set and the business can account for the missing interval. Those may be very different times.

Snapshot guidance is where shared responsibility becomes concrete

Google's status update during the incident told affected customers they could restore from snapshots. That recommendation is useful only for customers that had usable snapshots. A snapshot that does not exist, is too old, is in the wrong location, lacks application consistency, or has never been tested is not a recovery path. The incident therefore turned a common cloud phrase, shared responsibility, into a concrete question: who had actually created and verified a recovery point before the physical event?

Google's current data protection options for disks and instances guide frames recovery around recovery time objective, recovery point objective, use case, and cost. The snapshot creation documentation explains standard and archive snapshots. The snapshot overview describes incremental snapshots. The scheduled snapshots guide recommends schedules as a backup practice, and the snapshot best-practices page adds practical constraints and reliability advice. That current documentation is clearer than many early cloud-era assumptions.

Application consistency remains a customer concern. A disk snapshot captures block state; a database may need quiescing, flushing, or coordinated backup operations to make the restored state usable. Google's application-consistent Linux snapshot documentation explains snapshot schedules with guest flush. The important point is not the exact feature set in 2015 versus now. It is the durable control principle: recoverability requires a backup process aligned to the application, not merely a provider storage promise.

Small teams are especially exposed to this gap. A startup or municipal project may choose a single cloud zone to reduce latency and cost. It may run a database on a Persistent Disk and rely on the product name and provider reputation as a substitute for backup design. It may not have a dedicated storage engineer, a tested restore process, or a business impact analysis. The 2015 incident shows why documentation and product defaults matter: customers with less internal expertise need storage options and warnings that make the failure-domain boundary obvious.

Provider and customer duties should be stated in operational language. Google should design the storage system to survive expected physical hazards, publish clear failure-domain information, provide snapshot and replication tools, preserve incident evidence, and identify affected resources. The customer should select a recovery objective, schedule backups, validate restores, place snapshots or replicas outside the relevant failure domain, and decide whether locality constraints allow off-zone or off-region copies. Neither side can do the other's full job.

Regional disks and replication change the failure model, not the need for recovery thinking

Google now offers regional Persistent Disk and Hyperdisk high-availability options. The regional disk documentation explains disks replicated between zones in a region for higher availability, and the regional disk failover guide describes force attach when a primary zone fails. Google's blog on regional Persistent Disks for highly available workloads makes the availability use case explicit.

Those features are meaningful improvements for many workloads, but they do not eliminate architecture judgment. Regional replication can protect against zonal unavailability or storage errors in one zone. It may not protect against application-level corruption that is replicated, customer deletion, compromised credentials, a region-wide control problem, or a recovery point that is too recent to be useful. A customer still needs backups for corruption, retention, and rollback. A replicated disk is a high-availability mechanism; it is not automatically a complete data-protection program.

The same caution applies to snapshots. A snapshot can be independent of the failed disk and can be restored to another zone. It may still be too old, application-inconsistent, unavailable to the right project, encrypted with a key the recovery environment cannot access, or stored in a location that conflicts with policy. Google's disk encryption documentation reminds customers that disks and snapshots can involve different key choices. Backup strategy must include access, keys, retention, location, and restore tests, not only the existence of a snapshot entry.

Compute Engine's current SLA and the historical 2015 SLA version show another distinction. SLAs address service availability and credits under defined conditions. They are not a complete statement of recoverability or business loss. A credit can compensate a fraction of service charges while the customer still must restore data, reconcile transactions, notify users, or rebuild trust. The fact that the status page told affected customers to restore from snapshots shows that operational recovery lived outside the SLA credit question.

For data-sovereignty owners, replication choices require careful policy work. A customer may require data to remain in Europe or Belgium. That does not mean all copies must sit in one zone. It may allow snapshots in a European multi-region or another European region, depending on service terms, regulator expectations, and risk appetite. Conversely, a strict location requirement may prevent some cross-region backups and require a higher local availability design. The accountable act is to make that tradeoff explicit before data is lost.

Customer recovery evidence is part of the incident

Provider incident reports often end at service restoration. Data-loss events need a second ledger: customer recovery evidence. Which disks had read errors? Which writes were unrecoverable? Which customers restored from snapshots? Which snapshots failed or were too old? Which applications needed manual reconciliation? Which customer workloads had no backup? Which messages were sent to customers about permanent loss and workaround steps? Some of that evidence is private, but the categories matter publicly.

The status page's repeated impact percentages were useful because they avoided vague reassurance. Less than 1 percent susceptible, less than 0.1 percent with read failures, and less than 0.000001 percent permanent loss describe narrowing categories. They should not be merged into a single statement. Susceptible disks, actively failing disks, and unrecoverable data are different states. A customer in each state needs a different action.

The customer also needs resource-specific notice. A general status page tells the market that something is wrong. It does not tell one database operator whether a specific disk is affected. Google had the strongest ability to identify affected resources, correlate storage systems, and provide account-level notices. Customers had the strongest ability to check application consistency, restore from their own snapshots, and decide what recent business data might be missing. Both kinds of evidence are necessary.

This division is especially important for auditors. An auditor reviewing a cloud workload after such an event should not ask only whether the provider reported a small percentage. The right questions are whether the organization knew its recovery point objective, whether snapshots existed before the incident, whether restore tests had passed, whether backup locations matched policy, whether application owners accepted the residual loss, and whether the provider's notice supplied enough detail to classify the affected resources. If the answer is no, the failure was not only a provider incident; it was also an architecture governance gap.

Procurement should ask the same questions in advance. What failure domain does this disk occupy? What independent copy exists? Who owns snapshot schedules? How are restores tested? What is the maximum tolerable lost write interval? Does locality policy permit a replica elsewhere? What notice will the provider give if storage media, power, or control systems threaten data durability? What is the support path during a data-loss event? These questions turn "cloud durability" from a slogan into a risk decision.

Procurement should not buy a region as if it were a backup

The Belgium incident is especially useful for procurement because it exposes a common shortcut. A buyer asks where the data will live. The provider answers with a region or zone. The buyer treats that answer as resilience. But the location answer and the recovery answer are different contract questions. One describes placement; the other describes what happens after loss, corruption, or unavailability. A contract that secures data locality but leaves the backup design undefined has solved only half the problem.

A strong procurement record would identify the workload's required recovery point and recovery time before choosing storage. A logging workload may tolerate some delay but not silent loss. A transactional database may need application-consistent backups every few minutes. A public register may need immutable backups and tested restores. A small analytics project may accept daily snapshots. The storage product, snapshot schedule, replica location, encryption key design, and restore exercise should follow the mission requirement, not the other way around.

Procurement should also require a provider notice model. During a storage incident, the provider may know that a disk is in the affected population before the customer can diagnose it from application errors. The contract or support plan should specify how affected resources are identified, how customers are told whether a restore is recommended, how permanent loss is reported, how logs are preserved, and how technical support is prioritized. A generic service-status page is not enough for a data-loss event because the customer's action is resource-specific.

The buyer should also avoid a false choice between sovereignty and resilience. For many European workloads, an independent copy in another European region may satisfy policy while reducing single-zone risk. For stricter workloads, regional replication inside a country or carefully governed backup locations may be necessary. For some data, the cost and complexity of extra copies may be disproportionate. The accountability point is not that every workload needs the same design. It is that the tradeoff should be documented and accepted by the business owner who understands the consequence of lost writes.

Auditors should be wary of checklist answers. "Data is stored in Europe" does not answer whether it can be restored. "Persistent Disk is durable" does not answer whether the application can tolerate a recent-write loss. "Snapshots are available" does not answer whether they were configured, recent, complete, and tested. "The provider has an SLA" does not answer whether the customer has a usable copy. The audit evidence should include restore-test results, backup age, backup location, key access, and a record of who accepted the residual risk.

Small teams need defaults that make recoverability visible

The customers most likely to misunderstand the boundary are often the least equipped to recover from crossing it. Large enterprises may have storage teams, backup platforms, audit committees, and tabletop exercises. Small teams may have one engineer, one project, one region, and a dashboard that makes the disk look durable because the virtual machine can be deleted without deleting the volume. Their risk is not ignorance in a pejorative sense; it is the normal consequence of abstraction doing its job too well.

Cloud providers can reduce that risk through defaults and warnings. When a customer creates a single-zonal disk for a database-shaped workload, the interface can ask about backup schedules, recommend snapshot policies, show the failure domain, and warn that snapshots are required for independent recovery. Documentation can put failure-domain tables near creation workflows rather than deep in reliability guides. Pricing pages can show the cost of no backup as a risk acceptance, not only the cost of a snapshot as an extra.

Customers can reduce the risk with simple routines. Every persistent data store should have a named owner, a recovery point objective, a snapshot or backup schedule, a restore test date, a backup location, and a key-access plan. The first restore test should happen before production launch, not during the first incident. The test should restore to a separate environment, check application consistency, and confirm that the team can authenticate, decrypt, and reconnect the workload. If the team cannot afford the recovery design, that should be a conscious business decision.

The 2015 event is a good teaching case because the loss was not spectacular. There was no global collapse to make the lesson unavoidable. The percentage was tiny. Yet a small team with one affected disk and no recent snapshot could still face permanent loss. Resilience education often focuses on giant disasters; this incident shows that rare, narrow failures are enough to punish untested backup assumptions.

The same logic applies to internal platforms built by enterprises. A corporate platform team may offer "approved cloud templates" to product teams. Those templates should not simply create a zonal disk and leave backup choices to application owners who may not understand the storage layer. The platform should require or strongly guide snapshot schedules, replication options, retention periods, and restore testing. Shared responsibility inside a company mirrors shared responsibility with the cloud provider.

Data loss changes the moral weight of status language

Many outage statuses can be written in terms of elevated errors, degraded performance, or restoration. Data loss requires a different vocabulary. Customers need to know whether data is delayed, unavailable, corrupted, rolled back, partially unrecoverable, or permanently lost. Those categories produce different duties. Delayed data may require queue processing. Unavailable data may require failover. Corrupted data may require validation and rollback. Permanent loss may require notification, reconciliation, compensation, or legal review.

Google's status page moved carefully through read errors, degraded performance, and snapshot workaround. Contemporary reports later recorded permanent loss for a tiny storage fraction. The narrowing of affected populations was useful, but the public lesson is that permanent loss should be named clearly once known. A status page that stays with availability language too long can leave customers treating a data-loss event as a retry problem. A status page that names permanent loss too broadly can cause unnecessary panic. Precision is therefore not decorative; it controls customer response.

Good status language for storage incidents should state the affected product, zone, time range, resource class, symptom, current customer action, and evidence status. It should separate resources at risk from resources known to have read failures and resources with confirmed unrecoverable data. It should say whether snapshots, new disks, regional disks, or other storage products are affected. It should state whether the provider can identify affected resources directly and how customers will be contacted. It should update when the provider moves from service repair to data reconciliation.

This precision also helps customers report to their own stakeholders. A data-protection officer, auditor, board, or small-business owner needs to know whether the event changed confidentiality, integrity, availability, or recoverability. The 2015 event was availability plus recoverability for a narrow disk population. It was not evidence of unauthorized access. Treating every cloud incident as a breach is wrong; treating every storage incident as a transient availability issue is also wrong. The categories should fit the facts.

Locality decisions should include an exit story

Every locality decision should include an exit story: if this zone, region, or local storage choice fails, where does the workload go and what data follows it? A customer choosing europe-west1-b in 2015 needed to know whether a failed disk could be restored in another zone, whether the snapshot existed outside the failed system, whether the application could attach the restored disk, and whether DNS, credentials, and operators could bring the service back. Those questions remain current even though product names and features have changed.

An exit story has several parts. The first is data: what copy exists, how old it is, and where it lives. The second is compute: what environment can run the restored data. The third is identity and keys: who can access and decrypt it. The fourth is network and routing: how users reach the recovered service. The fifth is validation: how the team knows the restored application is correct. The sixth is communication: how users and stakeholders are told what happened and what data interval may be missing.

Locality constraints make the exit story more complex, but not optional. If data must stay in Belgium, the design may require multi-zone local resilience, more frequent snapshots, and stronger on-site backup controls. If data may stay within Europe, the design may use another European region or multi-region snapshot storage. If policy permits a global backup for disaster recovery, the design must still handle privacy, encryption, and access controls. The key is to decide explicitly rather than allowing the default disk placement to decide silently.

The provider can make this easier by presenting failure domains in customer language. Instead of only product names, the interface can describe "survives VM deletion," "survives zonal hardware failure class," "survives zone outage through regional replication," and "supports point-in-time restore through snapshots." No short phrase will cover every edge, but plain failure-domain language is harder to misread than broad durability adjectives.

Unknowns and careful boundaries

The public record does not name every affected customer, every lost write, every internal hardware model, or every post-incident engineering change. It does not prove that a specific customer's backup failure caused its loss. It does not establish a legal breach, negligence finding, damages award, or compliance violation. It also does not prove that all current Google Cloud storage products carry the same 2015 risk. Cloud infrastructure and product features have changed substantially since then.

The public record does support several firm conclusions. The event affected Persistent Disks in europe-west1-b. Customers experienced read errors. Google directed affected customers to restore from snapshots. Contemporary reports based on Google's account described four successive lightning strikes on the local utility grid, a brief power loss to storage systems, hardware susceptibility in a subset of storage, and permanent loss of a tiny fraction of allocated Persistent Disk space. Google said it would review the stack and upgrade storage hardware.

Current Google documentation makes snapshots, scheduled backups, regional disks, and data-protection choices explicit.

The most important inference is supported but should be marked as inference: clearer failure-domain disclosure and tested independent backups reduce the chance that a physical-site hazard becomes permanent application loss. That is not the same as saying every customer without a snapshot was negligent or that Google failed a legal duty. It is a practical control conclusion. The provider can make the boundary visible and build more resilient infrastructure. The customer can choose a recovery design that does not rely on one local disk.

The incident also cautions against two opposite mistakes. The first is cloud fatalism: concluding that because a hyperscale provider lost a tiny amount of data once, cloud storage cannot be trusted. The second is cloud complacency: concluding that because the percentage was tiny, no one needs independent backups. The mature position is more demanding and more useful. Use provider durability, but do not confuse it with a recovery point. Use locality, but do not confuse it with resilience. Use SLAs, but do not confuse credits with restored state.

The practical evidence test is simple enough to run before procurement closes. A buyer should be able to point to the live disk, the latest independent recovery point, the restore target, the identity and key path, the person accountable for a restore, and the most recent successful test. A provider should be able to point to the failure domain, the resource-specific notice path, the incident-status categories, and the support path for customers facing permanent loss. If either side cannot answer those questions before a rare storage event, the architecture is relying on hope hidden inside respectable product names.

That is why a small 2015 event still belongs in a 2026 accountability program. It turns an abstract shared-responsibility model into a visible operating contract. The provider's stack may be stronger now, and customers have more tools, but the decision logic remains the same: locality must be paired with a recoverable copy, a recoverable copy must be tested, and a tested recovery must be understood by the business owner who will face users when data is missing.

The final owner of that decision should not be hidden inside infrastructure shorthand. It should be a named service owner who understands the cost of a lost hour, a lost day, or a lost transaction interval.

That owner should also have authority to fund the backup.

Typography is the art and technique of arranging type to make written language legible, readable, and visually appealing. It involves selecting typefaces, point sizes, line lengths, line-spacing, and letter-spacing.

Typography originated with the invention of movable type by Johannes Gutenberg in the 15th century.
Key elements include font selection, kerning, tracking, and leading.
Good typography enhances readability and conveys mood or tone in design.

The Belgium disk-loss record remains useful because it is small enough to study and serious enough to change practice. It shows that provider-managed redundancy can fail at the edge of a physical hazard, that status percentages must be read by category, that snapshots matter only if they exist and restore cleanly, and that locality is not a substitute for independent recoverability. Google controlled the data-centre and storage stack; customers controlled their recovery architecture; auditors and procurement teams controlled whether those two responsibilities were examined before the next rare event.

The accountable result is a storage plan that can say where data lives, where a recoverable copy lives, how fresh it is, who can restore it, and what evidence will prove recovery when the local zone no longer can.

Typography

Typography originated with the invention of movable type by Johannes Gutenberg in the 15th century.
Key elements include font selection, kerning, tracking, and leading.
Good typography enhances readability and conveys mood or tone in design.