Spain public probe was facing issues and had intermittent failures between June 21st 20:00 UTC to June 22nd 8:40 UTC. Synthetic monitoring checks using Spain public probe have been failing intermittently during that window. The issue is now resolved - the probe doesn't suffer any issues anymore and is stable.
This incident has been resolved.
This incident has been resolved. No further issues were seen since adjusting the backend configuration on Friday June 20th. (22:22 UTC)
The root cause has been identified as node CPU saturation, causing high latency on ingesters.
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
There was a read outage impacting Loki tenants in the prod-us-east-0 cluster. This issue has been resolved.
This incident has been resolved.
Between 9:40 to 10:55 AM UTC, Cloud Logs service briefly experienced issue with providing data to query rate metrics only. You may experience gaps in the results for query rates panel in the billing dashboards for the given period. The situation is now mitigated, we apologize for the inconvenience.
From 18:15 to 18:25 UTC, our prod-us-west-0 Loki cell experienced a period of degraded write performance. The issue resolved quickly without requiring manual intervention, and the system has remained stable since.
The Tempo service on cluster EU west experienced a traffic increase over the weekend, which caused an elevated error rate in Tempo's write path (ingestion). Our engineering team identified the root cause of the issue, and implemented measurements for palliating and resolving the problem.
Traces ingestion problems could have been experienced from 15:30 UTC on 13th until 19:30 UTC on 15th.
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.