Grafana rulers crash-looping on prometheus

Resolved

This incident has been resolved.

Update

We are in the process of rolling out the fix. Our engineers are monitoring the progress.

Investigating

We are investigating issues with the grafana-ruler service on prod-us-east-2 and prod-us-west-0 which are causing periodic crash conditions. A code fix is currently being deployed to mitigate this