Partial Mimir Write Outage

Resolved

This incident has been resolved.

Both read and write 5xx's and increased latency were experienced in the two periods: 23:56:15 to 00:32:45 UTC 00:55:30 to 01:36:15 UTC

Update

Customers should no longer experience issues.

We will continue to monitor and provide updates.

Investigating

We are continuing to investigate this issue.

Investigating

Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.

Update

As of 00:28 UTC, we have observed improvement with the partial write outage. Customers should no longer experience issues with metrics ingestion.

We will continue to monitor and provide updates.

Investigating

As of 23:57 UTC, our engineers became aware of an issue with prod-us-west-0 resulting in a partial write outage. Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data.

We continue to investigate and will update when we have more to share.