February 08, 2023 23:38 UTC
|
WARN
|
ongoing
|
Datacenter Cooling Event - Southeast Asia - Extended Mitigation
|
See more
Start Time
February 08, 2023 23:38 UTC
|
Type
WARN
|
Affected Components
Southeast Asia - Azure Site Recovery
Southeast Asia - Virtual Machine Scale Sets
Southeast Asia - Azure Data Factory
Southeast Asia - Log Analytics
Southeast Asia - App Service
Southeast Asia - Web Apps
Southeast Asia - Azure NetApp Files
Non-Regional - Azure Active Directory (Azure AD)
Southeast Asia - Azure Backup
Southeast Asia - Azure IoT Central
Southeast Asia - Azure Machine Learning
Southeast Asia - Azure Stream Analytics
Southeast Asia - Storage Accounts
Southeast Asia - Virtual Machines
Southeast Asia - Web Apps Linux
|
Message
Datacenter Cooling Event - Southeast Asia - Extended Mitigation
|
Details
Impact Statement:
Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped a subset of cooling units offline in one of the Availability Zones. While working to restore the cooling units, temperatures in the datacenter increased and we have proactively powered down a small subset of compute and storage units to avoid damage to hardware and reduce cooling system load.
All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.
Current Status – 23:30 UTC
We are continuing to focus our efforts on mitigation for services which were impacted due to this incident. We have several key services which are fully recovered and some for which we are still working on post recovery checks. We are closely monitoring the datacenter metrics for storage and compute resources which continue to show healthy thresholds.
We will continue to provide updates as we have further progress towards service restoration. We will share our next update in 60 minutes or as events warrant.
This message was last updated at 23:36 UTC on 08 February 2023
|
|
February 08, 2023 12:23 UTC
|
WARN
|
about 11 hours
|
Datacenter Cooling Event - Southeast Asia
|
See more
Start Time
February 08, 2023 12:23 UTC
|
Type
WARN
|
Affected Components
Southeast Asia - Azure Video Indexer
Non-Regional - Azure Resource Manager
Non-Regional - Azure Migrate
Southeast Asia - Azure Cosmos DB
Southeast Asia - Web Apps
|
Message
Datacenter Cooling Event - Southeast Asia
|
Details
Impact Statement: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped a subset of the cooling units offline in one of the Availability zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units, to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.
Current Status: We do not have an exact ETA at this time, we currently expect an extended period of time to fully restore all cooling capacity. The Azure service recovery process will commence at this time and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require significant recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Customers using services that are zonally-aware may consider failing out of the impacted zone, physical zone AZ-03. Note that physical zones are mapped to logical zones in your Azure subscription, Refer to: https://learn.microsoft.com/rest/api/resources/subscriptions/check-zone-peers.
This message was last updated at 12:21 UTC on 08 February 2023
|
|
February 08, 2023 12:08 UTC
|
WARN
|
15 minutes
|
Datacenter Cooling Event - Southeast Asia
|
See more
Start Time
February 08, 2023 12:08 UTC
|
Type
WARN
|
Affected Components
Southeast Asia - Web Apps
Southeast Asia - Azure Cosmos DB
Non-Regional - Azure Resource Manager
Southeast Asia - Azure Video Indexer
|
Message
Datacenter Cooling Event - Southeast Asia
|
Details
Impact Statement: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped a subset of the cooling units offline in one of the Availability zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units, to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.
Current Status: We do not have an exact ETA at this time, we currently expect an extended period of time to fully restore all cooling capacity. The Azure service recovery process will commence at this time and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require significant recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Customers using services that are zonally-aware may consider failing out of the impacted zone, physical zone AZ-03. Note that physical zones are mapped to logical zones in your Azure subscription, Refer to: https://learn.microsoft.com/rest/api/resources/subscriptions/check-zone-peers.
This message was last updated at 12:06 UTC on 08 February 2023
|
|
February 08, 2023 11:58 UTC
|
WARN
|
10 minutes
|
Datacenter Cooling Event - Southeast Asia
|
See more
Start Time
February 08, 2023 11:58 UTC
|
Type
WARN
|
Affected Components
Southeast Asia - Azure Video Indexer
Southeast Asia - Web Apps
|
Message
Datacenter Cooling Event - Southeast Asia
|
Details
Impact Statement: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped a subset of the cooling units offline in one of the Availability zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units, to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.
Current Status: We do not have an exact ETA at this time, we currently expect an extended period of time to fully restore all cooling capacity. The Azure service recovery process will commence at this time and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require significant recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Customers using services that are zonally-aware may consider failing out of the impacted zone, physical zone AZ-03. Note that physical zones are mapped to logical zones in your Azure subscription, Refer to: https://learn.microsoft.com/rest/api/resources/subscriptions/check-zone-peers.
This message was last updated at 11:57 UTC on 08 February 2023
|
|
February 08, 2023 11:23 UTC
|
WARN
|
35 minutes
|
Datacenter Cooling Event - Southeast Asia
|
See more
Start Time
February 08, 2023 11:23 UTC
|
Type
WARN
|
Affected Components
Southeast Asia - Web Apps
Southeast Asia - Azure Video Indexer
|
Message
Datacenter Cooling Event - Southeast Asia
|
Details
Impact Statement: Starting around 20:19 UTC on 7 February 2023, a utility power surge in the Southeast Asia region tripped a subset of the cooling units offline in one of the Availability zones. While working to restore the cooling units, temperatures in the datacenter increased so we have proactively powered down a small subset of selected compute and storage scale units, to avoid damage to hardware and reduce cooling system load. All impacted storage and compute scale units are in the same datacenter, within one of the region’s three Availability Zones (AZs). Multiple downstream services have been identified as impacted.
Current Status: We do not have an exact ETA at this time, we currently expect an extended period of time to fully restore all cooling capacity. The Azure service recovery process will commence at this time and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require significant recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Customers using services that are zonally-aware may consider failing out of the impacted zone, physical zone AZ-03. Note that physical zones are mapped to logical zones in your Azure subscription, Refer to: https://learn.microsoft.com/rest/api/resources/subscriptions/check-zone-peers.
This message was last updated at 11:20 UTC on 08 February 2023
|
|
2024-04-23 23:28:16 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-23 23:28:16 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-23 23:28:16 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-23 23:28:16 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-23 23:28:16 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|