Summary

  • Google Cloud Storage’s 8 and 9 October incidents lasted 18 hours 11 minutes in total and were concentrated in us-central1 and us-central1-b; the reported customer symptoms centered on writes.
  • Google attributes the first episode to a routing optimization that bypassed capacity-aware logic and the recurrence to an incomplete rollback; the report names no affected customers or quantified losses.

A cloud storage outage is often described as a question of where copies live. Google’s incident report points to a different question: how a write request is steered through the service before it becomes a stored object. On 8 October, Cloud Storage customers using us-central1 saw higher write latency and errors for 12 hours 30 minutes. The issue returned the following day in the us-central1-b zone for 5 hours 41 minutes. Google’s combined duration is 18 hours 11 minutes, but this was two episodes with a recurrence, not one continuous 18-hour outage.

The public symptoms were specific. Google says customers primarily saw HTTP 503 UNAVAILABLE responses on WriteObject operations and client-side timeouts; successful writes were slower than usual. The current incident report says other Cloud Storage regions and other Google Cloud services were unaffected by this event. An earlier preliminary section mentioned minor effects on reads, but the final impact summary does not quantify read impact. The record therefore supports a bounded account of a write-path problem in one region and, during the recurrence, one zone.

It does not establish that every customer in the region was affected, that all writes failed, or that stored objects were lost.

Google’s final report describes the first trigger as a traffic-sorting configuration update intended to address local capacity constraints. The optimization bypassed the usual capacity-aware routing logic. A small fraction of backend nodes with little serving capacity then received a disproportionate share of write traffic. Those nodes exhausted resources; object-ingestion services began timing out; requests routed to them became slow or failed. A separate database issue added metadata-processing errors during part of the first window.

That is a layered causal account: a balancing decision changed request distribution, concentrated work, and exposed a second metadata fault alongside the main overload.

The next-day recurrence is the more consequential part of the operational record. Google attributes it to an incomplete rollback of an upload-planner feature that broke load balancing and overloaded smaller backend nodes. The first mitigation had not fully restored the previous safety condition. This does not show that every rollback is unreliable or that a broader regional failure occurred. It does show why incident closure and configuration state are different things: a service can appear to recover while a residual change leaves the same failure path available.

The recovery description also has a limit that operators should notice. Google initially tried to add emergency capacity, but says regional capacity limits blocked that option. Engineers instead drained overloaded nodes, disabled or reverted the optimization, rolled back a separate database update and redirected traffic. Once traffic and utilization stabilized, the drained nodes returned to service. Capacity is not always an emergency escape hatch; routing, traffic isolation and rollback accuracy may be the controls that remain available when a region cannot simply grow on demand.

Cloud Storage’s published architecture helps frame the event without overreading it. Google says a regional bucket stores data redundantly across at least two zones in its selected region. Dual- and multi-region buckets replicate across regions asynchronously after a write is first stored in one region across two zones. Google describes automatic serving from another region during a regional outage for those wider-location buckets, while warning that replication is asynchronous. Its consistency documentation says a successful object write is immediately readable.

These statements address redundancy, failover and acknowledgement semantics; they do not establish which bucket locations incident customers used.

That distinction is important. The incident report does not say that multi-region replication failed, that data was lost, or that geographic copies were unavailable. Instead, it documents a bottleneck in the path that accepted new writes. Regional redundancy can reduce exposure to a place-specific failure, but a design must still account for software and capacity controls shared by the write service. Storage resilience is not one property: durable copies, successful ingestion, metadata processing, read access and application recovery have different dependencies.

Google’s announced prevention work follows that chain. It says deployment protocols will validate load-balancing changes in smaller regions before larger, highly utilized regions; routing will enforce capacity constraints so traffic logic cannot bypass pre-spilling safeguards; and internal data movement will be improved in heavily used storage cells. These are future actions reported by Google, not independently verified completed controls. The public record does not provide test results, affected-request counts, customer names, bucket-location mix or measured losses.

The useful conclusion is narrower: this was a documented case where a routing optimization and a rollback weakness could defeat a local capacity safeguard twice, while geographic redundancy remained a separate question.

Sources: Google’s final incident report, Cloud Storage availability and durability, bucket locations, consistency, retry strategy.