• A Google Cloud network configuration update disrupted VMware Engine stretched clusters in Sydney, Melbourne and Frankfurt for nearly 12 hours
  • Compute and storage remained available, but failed links between cluster zones and the witness left some virtual machines isolated

The fact

A Google Cloud network configuration update disrupted VMware Engine stretched clusters in Sydney, Melbourne and Frankfurt on 14 July. The outage lasted 11 hours and 46 minutes. Google restored service by reversing the update and returning to the previous working configuration.

Compute and storage appeared to remain available, but communication between cluster zones failed. The affected zones also lost contact with the witness appliance, so the clusters could no longer synchronise safely. Some virtual machines became isolated and risked losing access to writable data. Google advised customers to contact support before moving affected workloads to the healthy side of a cluster.

The assessment

The incident affected customers using stretched clusters to keep workloads available across zones. Compute and storage continued running, but the network and witness connections needed to coordinate the clusters failed. VMware Engine stretched clusters place data nodes in two zones and use a witness in a third. When those components lost contact, some virtual machines became isolated and risked losing access to writable data. Google advised customers to consult support before moving affected workloads because the condition of the other zone was not always clear.

For BTW readers, the incident shows where operational responsibility still sits in a managed resilience service. Customers still rely on Google’s network and witness service to keep the clusters coordinated.Recovery plans therefore need to cover loss of that communication path, including when workloads can be moved safely and when provider support is required.

What to watch

Watch for Google’s final account of the outage and whether it leads to changes in network update procedures or clearer recovery guidance for stretched clusters that lose contact with the witness.