Tempo Write Path Failing
This incident has been resolved.
Incident History
Support Ticket Portal Unavailable
This incident has been resolved.
Severe degradation of certain tests
This incident has been resolved.
k6 test data processing severely degraded
This incident has been resolved.
Write path outage in us-central1 region
Due to this bug reported in https://github.com/kubernetes/kubernetes/issues/127370, we were affected by an issue causing K8S service endpoints not getting updated when pods are stopped/started if there are more than 1k pods matching the service. This caused a temporary outage in Mimir gossiping services, which further resulted in failures to ingest and query metrics for a short time. This issue has been resolved.
Issues with new stack creation
This incident has been resolved.
Adaptive Metrics Degraded Performance
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Grafana Cloud Portal Accessibility Issues
This incident has been resolved.
Degraded dashboard performance due to the erroneous security policy
Rollback has been completed as of 17:20 UTC. At this time, we are considering this issue resolved. No further updates.
Tempo Ingestion Disruption
This incident has been resolved.