Is Google Cloud Assured Workloads down?
Current Google Cloud Assured Workloads status is UP

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing
Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.
The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.
We do not have an ETA for replay of dropped notifications at this point.
We will provide more information by Tuesday, 2023-07-18 15:00 US/Pacific.
Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.
Workaround: None at this time.
Summary: Customers may see unexpected additional messages in GKE cluster logs.
Description: Mitigation work is currently underway by our engineering team.
We have identified a fix and tested it. The rollout of the fix started on Monday, 2023-07-10 and is expected to be complete by 2023-07-18. We will continue to provide updates on any status changes.
Customers may see unexpected additional messages in GKE cluster logs, for GKE clusters on version 1.24 and later. These additional messages should not have been added to the logs by GKE and can be safely ignored. Concerns for large workloads approaching quota have already been mitigated.
The workloads processed by GKE clusters are not impacted.
We will provide more information by Tuesday, 2023-07-18 13:00 US/Pacific.
Diagnosis: Customers may see unexpected additional log messages in GKE cluster logs.
Error/Warning log entries:
1. Failed to get record: decoder: failed to decode payload: EOF
2. cannot parse '0609 05:31:59.491654' after %L
3. invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'
Workaround: To avoid hitting Logging API quota customers can request quota Logging API increase.
To avoid storing these additional messages (that provides impact on logging billing) customers can apply Log exclusions by using the following query:
resource.labels.container_name = "fluentbit-gke" OR LOG_ID("fluentbit")
(jsonPayload.message =~ "^Failed to get record: decoder: failed to decode payload: EOF" OR jsonPayload.message =~ "^cannot parse .* after %L$" OR (jsonPayload.message =~ "^invalid time format" AND jsonPayload.plugin = "parser:glog"))

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing
Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.
The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.
We do not have an ETA for replay of dropped notifications at this point.
We will provide more information by Tuesday, 2023-07-18 15:00 US/Pacific.
Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.
Workaround: None at this time.
Summary: Customers may see unexpected additional messages in GKE cluster logs.
Description: Mitigation work is currently underway by our engineering team.
We have identified a fix and tested it. The rollout of the fix started on Monday, 2023-07-10 and is expected to be complete by 2023-07-18. We will continue to provide updates on any status changes.
Customers may see unexpected additional messages in GKE cluster logs, for GKE clusters on version 1.24 and later. These additional messages should not have been added to the logs by GKE and can be safely ignored. Concerns for large workloads approaching quota have already been mitigated.
The workloads processed by GKE clusters are not impacted.
We will provide more information by Tuesday, 2023-07-18 13:00 US/Pacific.
Diagnosis: Customers may see unexpected additional log messages in GKE cluster logs.
Error/Warning log entries:
1. Failed to get record: decoder: failed to decode payload: EOF
2. cannot parse '0609 05:31:59.491654' after %L
3. invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'
Workaround: To avoid hitting Logging API quota customers can request quota Logging API increase.
To avoid storing these additional messages (that provides impact on logging billing) customers can apply Log exclusions by using the following query:
resource.labels.container_name = "fluentbit-gke" OR LOG_ID("fluentbit")
(jsonPayload.message =~ "^Failed to get record: decoder: failed to decode payload: EOF" OR jsonPayload.message =~ "^cannot parse .* after %L$" OR (jsonPayload.message =~ "^invalid time format" AND jsonPayload.plugin = "parser:glog"))

Summary: Assured Workloads CreateWorkload API is experiencing intermittent issues while creating some workloads.
Description: We are experiencing an intermittent issue with Assured Workloads CreateWorkload API beginning at Monday, 2023-07-17 11:00 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2023-07-17 14:52 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers attempting to create workloads for some compliance regimes are experiencing intermittent failures. Only the compliance regimes where customer managed encryption keys is considered optional or required are currently affected.
Workaround: None at this time.
Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing
Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.
The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.
We do not have an ETA for replay of dropped notifications at this point.
We will provide more information by Monday, 2023-07-17 14:00 US/Pacific.
Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.
Workaround: None at this time.
Summary: Customers may see unexpected additional messages in GKE cluster logs.
Description: Mitigation work is currently underway by our engineering team.
We have identified a fix and tested it. The rollout of the fix started on Monday, 2023-07-10 and is expected to be complete by 2023-07-18. We will continue to provide updates on any status changes.
Customers may see unexpected additional messages in GKE cluster logs, for GKE clusters on version 1.24 and later. These additional messages should not have been added to the logs by GKE and can be safely ignored. Concerns for large workloads approaching quota have already been mitigated.
The workloads processed by GKE clusters are not impacted.
We will provide more information by Tuesday, 2023-07-18 13:00 US/Pacific.
Diagnosis: Customers may see unexpected additional log messages in GKE cluster logs.
Error/Warning log entries:
1. Failed to get record: decoder: failed to decode payload: EOF
2. cannot parse '0609 05:31:59.491654' after %L
3. invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'
Workaround: To avoid hitting Logging API quota customers can request quota Logging API increase.
To avoid storing these additional messages (that provides impact on logging billing) customers can apply Log exclusions by using the following query:
resource.labels.container_name = "fluentbit-gke" OR LOG_ID("fluentbit")
(jsonPayload.message =~ "^Failed to get record: decoder: failed to decode payload: EOF" OR jsonPayload.message =~ "^cannot parse .* after %L$" OR (jsonPayload.message =~ "^invalid time format" AND jsonPayload.plugin = "parser:glog"))

Summary: Assured Workloads CreateWorkload API is experiencing intermittent issues while creating some workloads.
Description: We are experiencing an intermittent issue with Assured Workloads CreateWorkload API beginning at Monday, 2023-07-17 11:00 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2023-07-17 14:52 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers attempting to create workloads for some compliance regimes are experiencing intermittent failures. Only the compliance regimes where customer managed encryption keys is considered optional or required are currently affected.
Workaround: None at this time.
Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing
Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.
The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.
We do not have an ETA for replay of dropped notifications at this point.
We will provide more information by Monday, 2023-07-17 14:00 US/Pacific.
Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.
Workaround: None at this time.
Summary: Customers may see unexpected additional messages in GKE cluster logs.
Description: Mitigation work is currently underway by our engineering team.
We have identified a fix and tested it. The rollout of the fix started on Monday, 2023-07-10 and is expected to be complete by 2023-07-18. We will continue to provide updates on any status changes.
Customers may see unexpected additional messages in GKE cluster logs, for GKE clusters on version 1.24 and later. These additional messages should not have been added to the logs by GKE and can be safely ignored. Concerns for large workloads approaching quota have already been mitigated.
The workloads processed by GKE clusters are not impacted.
We will provide more information by Tuesday, 2023-07-18 13:00 US/Pacific.
Diagnosis: Customers may see unexpected additional log messages in GKE cluster logs.
Error/Warning log entries:
1. Failed to get record: decoder: failed to decode payload: EOF
2. cannot parse '0609 05:31:59.491654' after %L
3. invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'
Workaround: To avoid hitting Logging API quota customers can request quota Logging API increase.
To avoid storing these additional messages (that provides impact on logging billing) customers can apply Log exclusions by using the following query:
resource.labels.container_name = "fluentbit-gke" OR LOG_ID("fluentbit")
(jsonPayload.message =~ "^Failed to get record: decoder: failed to decode payload: EOF" OR jsonPayload.message =~ "^cannot parse .* after %L$" OR (jsonPayload.message =~ "^invalid time format" AND jsonPayload.plugin = "parser:glog"))

Summary: Assured Workloads CreateWorkload API is experiencing intermittent issues while creating some workloads.
Description: We are experiencing an intermittent issue with Assured Workloads CreateWorkload API beginning at Monday, 2023-07-17 11:00 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2023-07-17 14:52 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers attempting to create workloads for some compliance regimes are experiencing intermittent failures. Only the compliance regimes where customer managed encryption keys is considered optional or required are currently affected.
Workaround: None at this time.
Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing
Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.
The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.
We do not have an ETA for replay of dropped notifications at this point.
We will provide more information by Monday, 2023-07-17 14:00 US/Pacific.
Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.
Workaround: None at this time.
Summary: Customers may see unexpected additional messages in GKE cluster logs.
Description: Mitigation work is currently underway by our engineering team.
We have identified a fix and tested it. The rollout of the fix started on Monday, 2023-07-10 and is expected to be complete by 2023-07-18. We will continue to provide updates on any status changes.
Customers may see unexpected additional messages in GKE cluster logs, for GKE clusters on version 1.24 and later. These additional messages should not have been added to the logs by GKE and can be safely ignored. Concerns for large workloads approaching quota have already been mitigated.
The workloads processed by GKE clusters are not impacted.
We will provide more information by Tuesday, 2023-07-18 13:00 US/Pacific.
Diagnosis: Customers may see unexpected additional log messages in GKE cluster logs.
Error/Warning log entries:
1. Failed to get record: decoder: failed to decode payload: EOF
2. cannot parse '0609 05:31:59.491654' after %L
3. invalid time format %m%d %H:%M:%S.%L%z for '0609 05:31:59.490647'
Workaround: To avoid hitting Logging API quota customers can request quota Logging API increase.
To avoid storing these additional messages (that provides impact on logging billing) customers can apply Log exclusions by using the following query:
resource.labels.container_name = "fluentbit-gke" OR LOG_ID("fluentbit")
(jsonPayload.message =~ "^Failed to get record: decoder: failed to decode payload: EOF" OR jsonPayload.message =~ "^cannot parse .* after %L$" OR (jsonPayload.message =~ "^invalid time format" AND jsonPayload.plugin = "parser:glog"))