A number of instances are unavailable or experiencing slow performance in the prod-eu-west-4
As of this time, we are considering this incident resolved. No further updates.
Incident History
Some Degraded Tempo Writes
We observed a brief spike in degraded Tempo writes for a small subset of users in the prod-eu-west-2 region, which have now been resolved.
k6 tests failing to start from CLI
The configuration change in our API gateway led to certain endpoints being unreachable. This impacted test runs executed from CLI as they were not able to start.
k6 cloud tests cannot start
This incident has been resolved.
Private Datasource Connect connectivity loss in 2 regions.
The incident is now resolved. An error in an infrastructure change meant that the SSH endpoint for PDC in eu-west-3 region was unavailable for customers to connect to. Reverting the change fixed the issue and PDC agents were able to reconnect successfully.
Frontend Observability app not found in some regions.
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Tempo Write Latency
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Issues affecting prod-us-central-0
An incident from our CSP, Google Cloud, caused network issues impacting our customers on prod-us-central-0. The Grafana Cloud incident resolved shortly after the CSP incident was resolved, by 6:00 UTC.
Broken CloudWatch Queries for Some Users
Engineering has released a fix and as of 23:30 UTC, customers should no longer experience broken CloudWatch queries.
At this time, we are considering this issue resolved. No further updates.
Loki-prod Write and Read Issues
This incident has been resolved.