Elevated Mimir Read/Write Errors

Resolved

We continue to observe a continued period of recovery. As of 00:08 UTC, we are considering this issue resolved. No further updates.

Update

A fix has been implemented and we are monitoring the results.

Investigating

Our teams have been alerted that Synthetic Monitoring will also be affected by this outage.

Users may see gaps in their Synthetic Monitoring metrics as well as missed alerts as a result of this.

We continue to investigate and will provide further updates as they become available.

Investigating

The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.

Investigating

The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.

Investigating

Span metrics have also been identified as affected. We are continuing to investigate the root cause and will share further updates as more information becomes available.

Investigating

We are investigating elevated Mimir read errors beginning at approximately 21:57 UTC.

The errors are technically retriable, but most retries are unlikely to succeed at this time. This may result in failed or delayed query responses for some users.

Engineering is actively investigating the root cause and working to restore normal read performance.

We will provide further updates as we learn more.