Tempo write degradation in prod-eu-west-3 - tempo-prod-08
Engineering has released a fix and we continue to observe a period of recovery. As of 15:12 UTC we are considering this resolved.
Incident History
Write Degradation in Grafana Cloud Logs (prod-us-east-3)
Between 20:23 UTC and 20:53 UTC, Grafana Cloud Logs in prod-us-east-3 experienced a write degradation, which may have resulted in delayed or failed log ingestion for some customers.
The issue has been fully resolved, and the cell is currently operating normally. We are continuing to investigate the root cause and will provide additional details if relevant.
Partial Write Outage in prod-us-central-0
There was a ~15 minute partial write outage for some customers in prod-us-central-0. The time frame for this outage was 15:43-15:57 UTC.
High Latency and Errors in Prod-Us-Central-7
This incident has been resolved.
Cloudflare Error 1016
From 20:32 to 20:37 UTC, a DNS record misconfiguration resulted in temporary Cloudflare 1016 DNS errors on many Grafana Cloud stacks.
The misconfiguration was mitigated within 5 minutes, and we are working with Cloudflare to better understand why the particular misconfiguration resulted in this outage.
K6 test-runs cannot be started and the overall navigation experience is degraded
This incident has been resolved.
PDC Queries in Prod-Us-East-3 Are Failing
This incident has been resolved.
New K6 Tests Intermittently Failing
This incident has been resolved.
Logs write path degradation on GCP Belgium - prod-eu-west-0
Today 17th from 10:00 UTC to 10:38 UTC we experienced logs write path degradation. Customers may have experienced 5xx errors while ingestion on that cluster. Service is fully restored.
IRM access issues on instances in GCP Belgium (prod-eu-west-0)
Due to an incident today in IRM (OnCall) product, access to this application was degraded from 10:39 UTC until 11:16. Customers may have experienced the app not operable or not accessible. The service is again fully restored.