Broken CloudWatch Queries for Some Users
Engineering has released a fix and as of 23:30 UTC, customers should no longer experience broken CloudWatch queries.
At this time, we are considering this issue resolved. No further updates.
Incident History
Loki-prod Write and Read Issues
This incident has been resolved.
Degraded browser synthetics performance in many probe clusters
We have addressed the problem and browser synthetics began recovering at 00:00 UTC. Browser checks on all probes have been operating normally since 00:15 UTC.
Issues with BigQuery Data Sources
This incident has been resolved.
Issues with BigQuery Data Sources
This incident has been resolved.
Widespread Outages in Prod-EU-West-0
This incident has been resolved.
Grafana Cloud Traces some trace search result unavailable in prod-ap-southeast-2 region.
Due to a configuration change, one cell in prod-ap-southeast-2 region was incorrectly configured causing some trace search results to not return as expected. The issue was fixed on July 28th around 12:20 UTC and all search results are now being returned as expected. No data was lost in the process.
IRM (OnCall and Incident) plugin failing to load in eu-west-2 and us-east-0 regions.
We confirmed no more issues since the resolution.
Some Users Unable to Connect New Datasources with PDC
This incident has been resolved.
New Pods Failing to Start for a Small Subset of Instances
At approximately 13:50 UTC a new deployment resulted in pointing to a non-existent plugin version for pinned instances. This resulted in new pods for those instances to fail upon start. Since older pods were still running, users should not have experienced a disruption.