Mimir Partial Write Outage

Resolved

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Update

Synthetic Monitoring has now also recovered. Customers should no longer experience alert rules failing to evaluate.

We continue to monitor for recurrence and will provide updates accordingly.

Update

Engineering has released a fix and as of 22:25 UTC, customers should no longer experience ingestion issues. We will continue to monitor for recurrence and provide updates accordingly.

Investigating

While investigating this issue, we also became aware that Synthetic Monitoring is affected. Some customers may have alert rules failing to evaluate.

Investigating

As of 21:30 UTC, we are experiencing a partial ingestion outage in Grafana Mimir. This is affecting the write path, where some ingestion requests are failing or timing out.

Our engineering team is actively investigating and working to identify the root cause.