Degraded read performance for Hosted Metrics prod-us-east-0 cluster.
At this time, we are considering this issue resolved since the read path has remained stable after the mitigation.
Incident History
Synthetic Monitoring Outage
On October 28, 2024, from 10:35 UTC to 11:25 UTC, our engineering team detected a major outage affecting Synthetic monitoring in the GCP US Central region.
During this time, users may have noticed gaps in their synthetic logs and metrics.
We’re happy to confirm that the issue has been resolved as of 11:25 UTC. Thank you for your patience.
Grafana Cloud having a datasource with PDC enabled using localhost address are getting 403 status responses.
The incident has been resolved.
MS SQL Data source unavailable for production instances on Daily or Instant RRC
Engineering has released a fix and as of 19:20 UTC, customers should no longer experience any issues with the MS SQL plugin. At this time, we are considering this issue resolved. No further updates.
Tempo Query Failures in AWS Clusters
The fix has been rolled out and issue has been reoslved.
Degraded Reads in Loki
At approximately ~11:30 UTC some customers in the prod-us-east-0 region experienced either extremely slow Loki queries, or Loki queries not returning at all. This lasted until ~ 15:30 UTC when the fix was rolled out.
Grafana OnCall degraded perfomance
This incident has been resolved.
k6: Operational issue
This incident has been resolved.
Slow access to OnCall UI and delayed notifications for prod-us-central-0 cluster
Engineering has released a fix and as of Thu Oct 3rd, around 23:00 UTC, customers should no longer experience delays while accessing OnCall UI. At this time, we are considering this issue resolved. No further updates.
Synthetic monitoring failures, London probe location
This incident has been resolved.