Incident History

This incident has been resolved.

This incident has been resolved.

We are investigating a partial read outage affecting Loki in prod-us-east-4. Between 19:41 UTC and 19:50 UTC, a significant portion of read queries may have failed or returned errors.

The issue has been identified and service has been restored. We are continuing to investigate the underlying cause and will provide additional information as it becomes available.

He have identified the cause and applied a fix for this issue and all effected stacks within the affected regions are not working as expected.

This incident has been resolved.

This incident has been resolved.

The issue affecting Grafana IRM in the EU region has been resolved. The IRM UI, public API, and alert notification processing have been restored and are operating normally.

Impact period: April 1 – July 29, 2026

Summary: During this period, some customers in a subset of regions may have experienced gaps in select ruler/recording rule metrics within their Grafana Cloud instances. Not all customers or environments in the listed regions were affected.

Affected regions: A subset of environments across the following regions may have been impacted: ap-south-1 au-southeast-1 ca-east-0 eu-central-0 eu-west-2 eu-west-7 us-west-0

Affected values: grafanacloud_instance_queries_per_second grafanacloud_instance_rule_config_last_reload_successful grafanacloud_instance_rule_evaluations_total:rate5m grafanacloud_instance_rule_evaluation_failures_total:rate5m grafanacloud_instance_rule_group_interval_seconds grafanacloud_instance_rule_group_last_duration_seconds grafanacloud_instance_rule_group_iterations_total:rate5m grafanacloud_instance_rule_group_iterations_missed_total:rate5m grafanacloud_instance_rule_group_last_evaluation_timestamp_seconds grafanacloud_instance_rule_group_rules grafanacloud_instance_ruler_queries_failed_total:rate5m grafanacloud_instance_ruler_queries_zero_fetched_series_total:rate5m grafanacloud_instance_ruler_notifications_sent_total:rate5m grafanacloud_instance_ruler_notifications_errors_total:rate5m grafanacloud_instance_ruler_notifications_queue_capacity grafanacloud_instance_ruler_notifications_queue_length grafanacloud_instance_ruler_notifications_latency_seconds:99quantile grafanacloud_instance_ruler_notifications_latency_seconds:50quantile

Current status: This issue has been resolved. No further action is required from customers. If you continue to notice gaps in the metrics described above, please reach out to support and reference this incident.

The issue affecting OTLP ingestion in the prod-us-east-3 region has been resolved. Between 13:30 UTC and 17:45 UTC, some customers experienced intermittent failures when writing telemetry to the OTLP endpoint. Logs were confirmed to be affected, and metrics and traces may also have experienced intermittent ingestion failures.

Our investigation has concluded, and the affected services have recovered.

This incident has been resolved.