Grafana rulers crash-looping on prometheus
This incident has been resolved.
This incident has been resolved.
We have found that the issues are related to the grafana.unifiedHomepage feature rollout active 16:50 UTC yesterday, to 10:20 UTC today. We have rolled this back and systems are now working as expected.
We identified an issue with the internal job that calculates month-to-date usage and cost data, which caused usage attribution and billing dashboards to display stale information. The root cause has been identified and resolved.
Your billing dashboard may show a sudden jump in usage. This is expected. It reflects several days of accumulated usage that hadn't been showing up while the issue was ongoing, not a sudden change in your actual usage.
This issue does not affect your end-of-month invoice. Your bill will be calculated based on actual usage, not the numbers displayed during this period.
We haven't noticed any further issues in this region for alert group processing since yesterday. This incident i fully resolved.
Grafana Cloud Logs in prod-eu-north-0 experienced a 10-minute partial write outage between 13:45 and 13:54 UTC. Impacted users may have experienced 5xx errors during this time.
This incident has been resolved. Thank you for your patience.
This incident has been resolved by restarting the affected services.
We’ve implemented a fix and can confirm the issue is fully resolved as of 20:25 UTC.
Thank you for your patience.
This incident has been resolved.
Since the mitigation has been applied, we have not seen the errors return. At this point, we are considering the incident resolved.