Issues with BigQuery Data Sources
This incident has been resolved.
Incident History
Widespread Outages in Prod-EU-West-0
This incident has been resolved.
Grafana Cloud Traces some trace search result unavailable in prod-ap-southeast-2 region.
Due to a configuration change, one cell in prod-ap-southeast-2 region was incorrectly configured causing some trace search results to not return as expected. The issue was fixed on July 28th around 12:20 UTC and all search results are now being returned as expected. No data was lost in the process.
IRM (OnCall and Incident) plugin failing to load in eu-west-2 and us-east-0 regions.
We confirmed no more issues since the resolution.
Some Users Unable to Connect New Datasources with PDC
This incident has been resolved.
New Pods Failing to Start for a Small Subset of Instances
At approximately 13:50 UTC a new deployment resulted in pointing to a non-existent plugin version for pinned instances. This resulted in new pods for those instances to fail upon start. Since older pods were still running, users should not have experienced a disruption.
IRM (OnCall + Incident) plugin not available
This incident has been resolved.
Degraded performance on Email notifications across all regions
Hosted Grafana experienced degraded performance on the Email notification service today, 15th of July from 11:45 UTC to 13:00 UTC. This impacted the sending/receiving of emails used in modules like alerting or reporting. Service was restored shortly after identification.
Most Prometheus Queries Failing in Prod-US-East-0
This incident has been resolved.
Grafana fast release channel error
This incident has been resolved.