February 14, 2024 19:35 UTC
|
DOWN
|
less than a minute
|
Multiple Google Cloud Products are experiencing issues in us-west1
|
See more
Start Time
February 14, 2024 19:35 UTC
|
Type
DOWN
|
Affected Components
Americas (regions) - Certificate Authority Service
Americas (regions) - Cloud Build
Americas (regions) - Cloud Run
Americas (regions) - Dialogflow CX
Americas (regions) - Google Cloud Deploy
Americas (regions) - Pub/Sub Lite
us-west1 - Certificate Authority Service
us-west1 - Cloud Build
us-west1 - Cloud Key Management Service
us-west1 - Cloud Run
us-west1 - Dialogflow CX
us-west1 - Google Cloud Deploy
us-west1 - Pub/Sub Lite
|
Message
Multiple Google Cloud Products are experiencing issues in us-west1
|
Details
Summary: Multiple Google Cloud Products are experiencing issues in us-west1
Description: We are experiencing an issue with multiple Google Cloud Products beginning on Wednesday, 2024-02-14 9:40 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Wednesday, 2024-02-14 12:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Existing customer load balancers will continue to function. New load balancers or changes to existing load balancers will not propagate configs and changes to the configurations of load balancers may result in an error.
Configuration changes can not be made to the Regional Internal, Regional External, and Global External Application Load Balancers in the affected region.
Customers may see errors when making configuration changes.
Workaround: None at this time.
|
|
February 14, 2024 19:35 UTC
|
DOWN
|
about 2 hours
|
Multiple Google Cloud Products are experiencing issues in us-west1
|
See more
Start Time
February 14, 2024 19:35 UTC
|
Type
DOWN
|
Affected Components
Americas (regions) - Certificate Authority Service
Americas (regions) - Cloud Build
Americas (regions) - Cloud Run
Americas (regions) - Dialogflow CX
Americas (regions) - Google Cloud Deploy
Americas (regions) - Pub/Sub Lite
us-west1 - Certificate Authority Service
us-west1 - Cloud Build
us-west1 - Cloud Key Management Service
us-west1 - Cloud Run
us-west1 - Dialogflow CX
us-west1 - Google Cloud Deploy
us-west1 - Pub/Sub Lite
|
Message
Multiple Google Cloud Products are experiencing issues in us-west1
|
Details
Summary: Multiple Google Cloud Products are experiencing issues in us-west1
Description: We are experiencing an issue with multiple Google Cloud Products beginning on Wednesday, 2024-02-14 9:40 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Wednesday, 2024-02-14 12:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Existing customer load balancers will continue to function. New load balancers or changes to existing load balancers will not propagate configs and changes to the configurations of load balancers may result in an error.
Configuration changes can not be made to the Regional Internal, Regional External, and Global External Application Load Balancers in the affected region.
Customers may see errors when making configuration changes.
Workaround: None at this time.
|
|
November 09, 2023 01:30 UTC
|
WARN
|
less than a minute
|
Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
See more
Start Time
November 09, 2023 01:30 UTC
|
Type
WARN
|
Affected Components
Europe (regions) - Cloud Run
Asia Pacific (regions) - Cloud Run
Europe (regions) - Google App Engine
Asia Pacific (regions) - Google App Engine
Europe (regions) - Google Cloud Functions
Asia Pacific (regions) - Google Cloud Functions
Europe (regions) - Secret Manager
Asia Pacific (regions) - Secret Manager
europe-west1 - Cloud Run
europe-west1 - Google App Engine
europe-west1 - Google Cloud Functions
australia-southeast1 - Cloud Run
australia-southeast1 - Google App Engine
australia-southeast1 - Google Cloud Functions
australia-southeast1 - Secret Manager
europe-west1 - Secret Manager
|
Message
Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
Details
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
A rollout containing a fix to no longer ingest the excess logs to Cloud Logging is about 25% complete.
We will provide an update by Friday, 2023-11-10 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above.
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.
|
|
November 09, 2023 01:30 UTC
|
WARN
|
3 months
|
Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
See more
Start Time
November 09, 2023 01:30 UTC
|
Type
WARN
|
Affected Components
Europe (regions) - Cloud Run
Asia Pacific (regions) - Cloud Run
Europe (regions) - Google App Engine
Asia Pacific (regions) - Google App Engine
Europe (regions) - Google Cloud Functions
Asia Pacific (regions) - Google Cloud Functions
Europe (regions) - Secret Manager
Asia Pacific (regions) - Secret Manager
europe-west1 - Cloud Run
europe-west1 - Google App Engine
europe-west1 - Google Cloud Functions
australia-southeast1 - Cloud Run
australia-southeast1 - Google App Engine
australia-southeast1 - Google Cloud Functions
australia-southeast1 - Secret Manager
europe-west1 - Secret Manager
|
Message
Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
Details
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
A rollout containing a fix to no longer ingest the excess logs to Cloud Logging is about 25% complete.
We will provide an update by Friday, 2023-11-10 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above.
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.
|
|
November 08, 2023 23:55 UTC
|
WARN
|
less than a minute
|
Customer experiencing a requests to Secret Manager timeout for App Engine and Cloud Run. and Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
See more
Start Time
November 08, 2023 23:55 UTC
|
Type
WARN
|
Affected Components
Europe (regions) - Cloud Run
Asia Pacific (regions) - Cloud Run
Europe (regions) - Google App Engine
Asia Pacific (regions) - Google App Engine
Europe (regions) - Google Cloud Functions
Asia Pacific (regions) - Google Cloud Functions
Europe (regions) - Secret Manager
Asia Pacific (regions) - Secret Manager
europe-west1 - Cloud Run
europe-west1 - Google App Engine
europe-west1 - Google Cloud Functions
australia-southeast1 - Cloud Run
australia-southeast1 - Google App Engine
australia-southeast1 - Secret Manager
australia-southeast1 - Google Cloud Functions
europe-west1 - Secret Manager
|
Message
Customer experiencing a requests to Secret Manager timeout for App Engine and Cloud Run. and Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
|
Details
Summary: Customer experiencing a requests to Secret Manager timeout for App Engine and Cloud Run.
Description: Mitigation work is currently underway by our engineering team.
We do not have an ETA for mitigation at this point.
We will provide more information by Wednesday, 2023-11-08 16:40 US/Pacific.
Diagnosis: Requests to Secret Manager might time out for App Engine and Cloud Run customers.
Workaround: Some customers reported that removing the secret manager calls and "hard coded" the secrets is a workaround.
Summary: Google Kubernetes Engine customers with Workload Identity enabled may see high application logging rate
Description: There is no impact to cluster functionality or performance. The only impact is extra logs in Cloud Logging.
gke-metadata-server is a GKE-managed system workload that is part of the GKE Workload Identity feature. Versions 0.4.272 to 0.4.280 of gke-metadata-server contain an incorrect configuration that results in a high rate of debug logs that contain the string "Unable to sync sandbox". These logs are then ingested into Cloud Logging, consuming Cloud Logging ingestion quota, and causing excess billable usage when exceeding the free monthly allotment.
A rollout containing a fix to no longer ingest the excess logs to Cloud Logging is about 25% complete.
We will provide an update by Friday, 2023-11-10 14:00 US/Pacific.
Diagnosis: Customers can determine whether their cluster is impacted by inspecting the gke-metadata-server daemonset with kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and looking at the components.gke.io/component-version annotation in .spec.template.metadata.annotations. If the value is a version between 0.4.272 and 0.4.280 (inclusive), then the cluster is currently affected.
Workaround:
- Customers using GKE Rapid Channel can upgrade their cluster control plane to 1.28.2-gke.1157000 and above, or 1.27.7-gke.1038000 and above.
- Customers on GKE Regular Channel, GKE Stable Channel, or who are not using release channels do not have a workaround at this time.
|
|
2024-04-11 07:09:53 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-11 07:09:53 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-11 07:09:53 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-11 07:09:53 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|
2024-04-11 07:09:53 UTC UTC
|
STATUS
|
? minutes
|
Sign in to view more status history.
|
See more
|