Issue with Grafana Access in prod-ap-south-1 and prod-ap-northeast-0
We continue to observe a continued period of recovery. At this time, we are considering this issue resolved.
Incident History
Degraded performance when querying metrics on AWS Germany
On 12th of March, between 13:30 UTC and 13:50 UTC we experienced some degraded performance when querying metrics on the AWS Germany cluster. The effects were reflected in unavailability to see metrics on dashboards and alert queries triggering errors during that period. The issue has been resolved.
Delay in aggregated series through adaptive metrics
All series are now caught up and the incident is resolved.
Write path down in prod-us-east-2-prometheus-prod-56
This incident has been resolved.
Read errors in prod-eu-west-0
Many ingesters were evicted from nodes in cortex-prod-01 at once causing a read path outage. Once the ingesters were rescheduled the read path recovered. The errors lasted about 10 minutes.
Degraded performance due to overloaded internal queues
Our internal queues were affected by a scheduled data migration. For around three hours asynchronous scheduled tasks were affected and their processing delayed, but none cancelled or lost. Test scheduled particularly may have been not run at their intended time during that period.
Degraded UX with IRM Slack Integration.
This incident has been resolved.
New instance creation taking longer than expected
This incident has been resolved.
Longer than expected load times in multiple AWS regions
This incident has been resolved.
New Instances failing to create across all regions.
Our Engineering team has identified that this issue is related to: https://status.grafana.com/incidents/b6yc10yk8kv7 We'll be marking this incident as resolved and will provide updates for this event as well in the status page mentioned above.