Engineering has released a fix and as of 16:07 UTC, customers should no longer experience read/write path degradation on us-central2 cloud metrics and traces. At this time, we are considering this issue resolved. No further updates.
This incident has been resolved.
Engineering has released a fix and as of 10:30 UTC, customers should no longer experience hosted services issues. At this time, we are considering this issue resolved. No further updates.
All clusters have been restarted and datasources should be fully operational.
Starting at ~21:11 UTC, Loki had a brief outage. Users may have experienced degraded write path performance for approximately 10 minutes, and 2-3 minutes of total write and read outage. This has since been resolved.
This incident has been resolved.
This incident has been resolved.
This incident has been resolved.
This incident has been resolved.
The incident is fully resolved now, root cause was determined and mitigated. The particular cell affected had a legacy config present, which caused a panic during the regular rollout. The cell was rolled back as a precaution and the mentioned config is now corrected.