Connectivity issues for Azure PrivateLink endpoints.
The scope of this incident was smaller than originally anticipated.
As of 16:27 UTC our engineering team merged a fix for those affected and we are considering this as resolved.
Incident History
PDC Agent Connectivity Issues in prod-eu-west-3
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.
Tempo write degradation in prod-eu-west-3 - tempo-prod-08
Engineering has released a fix and we continue to observe a period of recovery. As of 15:12 UTC we are considering this resolved.
Write Degradation in Grafana Cloud Logs (prod-us-east-3)
Between 20:23 UTC and 20:53 UTC, Grafana Cloud Logs in prod-us-east-3 experienced a write degradation, which may have resulted in delayed or failed log ingestion for some customers.
The issue has been fully resolved, and the cell is currently operating normally. We are continuing to investigate the root cause and will provide additional details if relevant.
Partial Write Outage in prod-us-central-0
There was a ~15 minute partial write outage for some customers in prod-us-central-0. The time frame for this outage was 15:43-15:57 UTC.
High Latency and Errors in Prod-Us-Central-7
This incident has been resolved.
Cloudflare Error 1016
From 20:32 to 20:37 UTC, a DNS record misconfiguration resulted in temporary Cloudflare 1016 DNS errors on many Grafana Cloud stacks.
The misconfiguration was mitigated within 5 minutes, and we are working with Cloudflare to better understand why the particular misconfiguration resulted in this outage.
K6 test-runs cannot be started and the overall navigation experience is degraded
This incident has been resolved.
PDC Queries in Prod-Us-East-3 Are Failing
This incident has been resolved.
New K6 Tests Intermittently Failing
This incident has been resolved.