Prometheus writes in prod-eu-west-3 are degraded

Resolved

This incident has been resolved. Thank you for your patience.

Update

We are continuing to monitor for any further issues.

Update

We have deployed mitigation and seen improvement in write failures over the past week. We are still seeing intermittent spikes in latency and continue to monitor.

Update

We are still seeing intermittent issues and continue to seek a resolution

Update

We are continuing to monitor for any further issues.

Update

We are continuing to monitor this through the weekend.

Update

We are continuing to monitor the previously impacted environments.

Update

A fix has been implemented and we are monitoring the results.

Investigating

We are continuing to investigate this issue.

Investigating

The metric writes issue reported in https://status.grafana.com/incidents/gfshj17lxj5z is still ongoing.

Our Engineering team is actively investigating this and we will provide further updates as our investigation progresses.