Is Google Cloud Storage down?
Current Google Cloud Storage status is UP

Summary: Google BigQuery is experiencing issues with cross region ‘copy transfer’ in multiple regions.
Description: Mitigation work is still underway by our engineering team.
Current data indicates that the issue has been mitigated for asia-northeast1, multi-regions: eu, europe-west1, europe-west2, europe-west3, europe-west9, and europe-north1. Our engineering team is continuing to work on mitigating the issue for us-west2.
We do not have an ETA for mitigation at this point.
We will provide more information by Wednesday, 2023-11-08 12:30 US/Pacific.
Diagnosis: The impacted customers may experience errors and latency while performing cross region ‘copy transfer.
Workaround: The impacted customers may retry the failed transfer.
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
The root cause has been identified and we are starting the rollout of a fix across the fleet.
We will provide more information by Wednesday, 2023-11-08 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.

Summary: Google BigQuery is experiencing issues with cross region ‘copy transfer’ in multiple regions.
Description: Mitigation work is still underway by our engineering team.
Current data indicates that the issue has been mitigated for asia-northeast1, multi-regions: eu, europe-west1, europe-west2, europe-west3, europe-west9, and europe-north1. Our engineering team is continuing to work on mitigating the issue for us-west2.
We do not have an ETA for mitigation at this point.
We will provide more information by Wednesday, 2023-11-08 12:30 US/Pacific.
Diagnosis: The impacted customers may experience errors and latency while performing cross region ‘copy transfer.
Workaround: The impacted customers may retry the failed transfer.
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
The root cause has been identified and we are starting the rollout of a fix across the fleet.
We will provide more information by Wednesday, 2023-11-08 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.

Summary: Inter-continental user facing packet loss between Europe and North America
Description: We are experiencing an issue with Google Cloud Networking, Cloud Load Balancing, Google Cloud Storage.
Our engineering team continues to investigate the issue.
We will provide an update by Wednesday, 2023-11-08 09:45 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: None at this time
Workaround: None at this time
Summary: Google BigQuery is experiencing issues with cross region ‘copy transfer’ in multiple regions.
Description: Mitigation work is still underway by our engineering team.
Current data indicates that the issue has been mitigated for Multi-regions: eu, europe-west1, europe-west2, europe-west3 and europe-west9. Our engineering team is continuing to work on mitigating the issue for other affected regions.
We do not have an ETA for mitigation at this point.
We will provide more information by Wednesday, 2023-11-08 09:20 US/Pacific.
Diagnosis: The impacted customers may experience errors and latency while performing cross region ‘copy transfer.
Workaround: The impacted customers may retry the failed transfer.
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
The root cause has been identified and we are starting the rollout of a fix across the fleet.
We will provide more information by Wednesday, 2023-11-08 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.

Summary: Inter-continental user facing packet loss between Europe and North America
Description: We are experiencing an issue with Google Cloud Networking, Cloud Load Balancing, Google Cloud Storage.
Our engineering team continues to investigate the issue.
We will provide an update by Wednesday, 2023-11-08 09:45 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: None at this time
Workaround: None at this time
Summary: Google BigQuery is experiencing issues with cross region ‘copy transfer’ in multiple regions.
Description: Mitigation work is still underway by our engineering team.
Current data indicates that the issue has been mitigated for Multi-regions: eu, europe-west1, europe-west2, europe-west3 and europe-west9. Our engineering team is continuing to work on mitigating the issue for other affected regions.
We do not have an ETA for mitigation at this point.
We will provide more information by Wednesday, 2023-11-08 09:20 US/Pacific.
Diagnosis: The impacted customers may experience errors and latency while performing cross region ‘copy transfer.
Workaround: The impacted customers may retry the failed transfer.
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
The root cause has been identified and we are starting the rollout of a fix across the fleet.
We will provide more information by Wednesday, 2023-11-08 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.

Summary: us-central1 : Elevated latency in accessing Google Cloud Storage
Description: We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2023-09-07 08:19 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Thursday, 2023-09-07 12:30 US/Pacific with current details.
Diagnosis: Customers in the impacted region may experience elevated latency.
Workaround: None at this time.
Summary: We are investigating an Issue with Vertex AI Search
Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.
Engineers are now working to clean up indexes that were created for non-canonical urls.
We do not have an ETA for full resolution at this point.
We will provide an update by Thursday, 2023-09-07 13:00 US/Pacific with current details.
Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.
There is the potential for blocked url indexing pipelines due to quotas being reached.
Additionally, searches may return duplicate results for the same webpage with different URLs.
Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.