Incident History

PDC Degraded performance

This incident has been resolved. Thank you for your patience.

Delayed ingestion and recording rule evaluation failures for Mimir in prod-ap-south-1

This incident has been resolved.

Loki read path in prod-eu-west-2 was down

Between 5:49 and 5:54 UTC, the read path in prod-eu-west-2 was down. This has completely recovered by 5:57 UTC. Recording rules may have failed to evaluate during this period which may result in gaps.

Loki Billing Data Lost

Between 13:57 and 14:20 UTC (23 minutes), a data gap occurred affecting billing usage data for Loki. Data for this window was not recorded and cannot be recovered.

Grafana rulers crash-looping on prometheus

This incident has been resolved.

Cannot access dashboard set up as a home page

We have found that the issues are related to the grafana.unifiedHomepage feature rollout active 16:50 UTC yesterday, to 10:20 UTC today. We have rolled this back and systems are now working as expected.

Delayed usage and billing data

We identified an issue with the internal job that calculates month-to-date usage and cost data, which caused usage attribution and billing dashboards to display stale information. The root cause has been identified and resolved.

Your billing dashboard may show a sudden jump in usage. This is expected. It reflects several days of accumulated usage that hadn't been showing up while the issue was ongoing, not a sudden change in your actual usage.

This issue does not affect your end-of-month invoice. Your bill will be calculated based on actual usage, not the numbers displayed during this period.