Incident History

Issue with Dashboard Views Being Registered

This is related to https://status.grafana.com/incidents/rhrk2ck6ly0y which was resolved by mistake.

For some dashboards, the Views/Error counts in the Dashboards/Folder list renders as - and never updates, even after dashboards are viewed repeatedly. This is ultimately causing inaccurate or missing view counts.

We are continuing to investigate the underlying cause of this issue. At this time, the scope and impact remain unchanged, and we have no new information to share. We will provide another update as soon as more information becomes available.

1784016645 Ongoing

Network Degredation in prod-us-central-0

From approximately 20:24 UTC - 20:47 UTC a network issue in prod-us-central-0 isolated part of our infrastructure in one availability zone, temporarily cutting off connectivity to a subset of backend services. This caused elevated query latency and some delayed metric evaluations, along with related write-path errors in a logging subsystem. All systems recovered automatically once network connectivity was restored.

Impacted customers may have had queries or rule evaluations fail during the incident, but ingestion was not impacted.

1783719114 - 1783719114 Resolved

IRM Mobile App forcing some users to logout

This incident has been resolved.

1783715252 - 1783947634 Resolved

Issue with Dashboard Views Being Registered

At this stage, we are considering the incident resolved.

1783699069 - 1783920189 Resolved

PDC Degraded performance

This incident has been resolved. Thank you for your patience.

1783597025 - 1783626870 Resolved

Delayed ingestion and recording rule evaluation failures for Mimir in prod-ap-south-1

This incident has been resolved.

1783592891 - 1783602462 Resolved

Loki read path in prod-eu-west-2 was down

Between 5:49 and 5:54 UTC, the read path in prod-eu-west-2 was down. This has completely recovered by 5:57 UTC. Recording rules may have failed to evaluate during this period which may result in gaps.

1783582631 - 1783582631 Resolved

Loki Billing Data Lost

Between 13:57 and 14:20 UTC (23 minutes), a data gap occurred affecting billing usage data for Loki. Data for this window was not recorded and cannot be recovered.

1783524803 - 1783524803 Resolved

Grafana rulers crash-looping on prometheus

This incident has been resolved.

1783513036 - 1783531416 Resolved

Cannot access dashboard set up as a home page

We have found that the issues are related to the grafana.unifiedHomepage feature rollout active 16:50 UTC yesterday, to 10:20 UTC today. We have rolled this back and systems are now working as expected.

1783506507 - 1783508443 Resolved
⮜ Previous