Tempo Write Latency
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Incident History
Issues affecting prod-us-central-0
An incident from our CSP, Google Cloud, caused network issues impacting our customers on prod-us-central-0. The Grafana Cloud incident resolved shortly after the CSP incident was resolved, by 6:00 UTC.
Broken CloudWatch Queries for Some Users
Engineering has released a fix and as of 23:30 UTC, customers should no longer experience broken CloudWatch queries.
At this time, we are considering this issue resolved. No further updates.
Loki-prod Write and Read Issues
This incident has been resolved.
Degraded browser synthetics performance in many probe clusters
We have addressed the problem and browser synthetics began recovering at 00:00 UTC. Browser checks on all probes have been operating normally since 00:15 UTC.
Issues with BigQuery Data Sources
This incident has been resolved.
Issues with BigQuery Data Sources
This incident has been resolved.
Widespread Outages in Prod-EU-West-0
This incident has been resolved.
Grafana Cloud Traces some trace search result unavailable in prod-ap-southeast-2 region.
Due to a configuration change, one cell in prod-ap-southeast-2 region was incorrectly configured causing some trace search results to not return as expected. The issue was fixed on July 28th around 12:20 UTC and all search results are now being returned as expected. No data was lost in the process.
IRM (OnCall and Incident) plugin failing to load in eu-west-2 and us-east-0 regions.
We confirmed no more issues since the resolution.