Google and Oracle’s cloud servers suffer outage during UK heat wave • The Register


Cloud services and servers hosted by Google and Oracle in the UK have been taken offline due to cooling issues as the country experiences a record heat wave.

When the mercury reached 40.3C (104.5F) in the east of England, the highest ever recorded by a country unaccustomed to such conditions, data centers could not withstand the heat. Selected machines were powered off to prevent long-term damage, causing certain resources, services and virtual machines to become unavailable, unlucky websites to be taken down, and more.

Several Oracle Cloud Infrastructure resources are offline, including networking, storage, and compute provided by its servers in the south of the UK. Cooling systems were blamed and technicians shut down equipment in an effort to prevent hardware from burning out, according to an update from Team Oracle.

Advertising

“Due to unseasonable temperatures in the region, a subset of the cooling infrastructure in the UK South (London) data center experienced an issue,” Oracle said Tuesday at 1638 UTC. “As a result, some customers may not be able to access or use Oracle Cloud Infrastructure resources hosted in the region.

“Affected service teams have been engaged and are working to restore the affected infrastructure to a healthy state, but as a precaution, we are in the process of identifying service infrastructure that can be safely de-energized to avoid other hardware failures. This measure is taken with the aim of limiting the potential for long-term impact on our customers. »

We are told that at least part of Oracle’s cooling infrastructure went down around lunchtime UK time.

OVH_server_rack_liquid_cooling

Your Top 5 Liquid Cooling Dilemmas Answered

READ MORE

Oracle isn’t the only IT giant to report temperature-related outages. Google Cloud says a number of its products “experience high error rates, latencies, or service unavailability” when served from systems located in europe-west2-a, which is one of its London facilities.

These issues affect various storage and compute services, including BigQuery, SQL, and Kubernetes. Google acknowledged the downtime at 1615 UTC. This outage, on the one hand, took down WP Engine hosted WordPress websites in the UK, which were powered by Google Cloud.

“There was a cooling-related outage in one of our buildings that hosts the west-europe2-a zone for the west-europe2 region,” according to a separate Google advisory.

“This caused a partial capacity failure in this area, resulting in VM terminations and loss of machines for a small number of our customers. We are working hard to bring the cooling back into service and build capacity in this area. We do not anticipate any additional impact in the europe-west2-a zone and running virtual machines should not be affected.

“In order to avoid machine damage and prolonged outage, we have powered down part of the area and are limiting preemptible GCE launches. We are seeing regional impact for a small proportion of newly launched persistent disk volumes and are working to restore redundancy for affected replicated persistent disk devices. »

The register asked Oracle and Google for further comment.

Extreme temperatures have also sparked fires in parts of England, affecting motorway traffic, rail services and electricity, with Luton Airport also temporarily closed due to melting runways. We’ll let you know if other Internet services are also affected. ®

Update to add

We’ve posted a follow-up here, in which we assess the consequences of cooling-related failures.

Leave a Comment