Read errors in prod-eu-west-0

Resolved

Many ingesters were evicted from nodes in cortex-prod-01 at once causing a read path outage. Once the ingesters were rescheduled the read path recovered. The errors lasted about 10 minutes.

Update

A fix has been implemented and we are monitoring the results.

Investigating

As of 3:35pm UTC, we were alerted to an issue with a Mimir read path outage in prod-01-eu-west-0. Users experiencing this issue may have encountered timeouts on Prometheus/metrics queries.

Service has recovered but we are monitoring.