Incident History

Missing Billing Metrics for Loki

An incident has impacted Loki billing metrics from 05:30 to 06:10 UTC across all clusters. This is currently resolved, however, users may notice some billing metrics missing from the billing dashboard during this time period. There is no impact of logs querying or ingestion.

1763533239 - 1763533239 Resolved

Hosted grafana is currently being impacted as a result of the Cloudflare outage

This incident has been resolved.

1763467158 - 1763487286 Resolved

Elevated Mimir Read/Write Errors

We continue to observe a continued period of recovery. As of 00:08 UTC, we are considering this issue resolved. No further updates.

1763418139 - 1763424603 Resolved

Metrics Write Outage in Multiple Cells

As of 23:07 UTC we are considering this incident as resolved.

Mitigation efforts have restored normal write performance, and error rates have returned to expected levels.

We have confirmed stability across the affected areas and continue to monitor, but no further impact is expected.

If you continue to experience any issues, please reach out to support.

1763409762 - 1763420920 Resolved

Hyderabad Probe Issues

We experienced degraded service with the Hyderabad probe today around 13:20 UTC which was resolved as of 17:30 UTC.

1763406104 - 1763406104 Resolved

Cortex - read/write path disruption

This incident has been resolved.

1763388393 - 1763450782 Resolved

PDC-Prod-eu-west-2 cluster degraded performance

This incident has been resolved.

1763380560 - 1763382181 Resolved

Loki Prod 012 read-path-unstable

resolved since 03:02UTC

1763352772 - 1763355349 Resolved

Degraded Brower Check Performance

Spanning from November 10th, 18:00 UTC to November 11th, 22:00 UTC, Synthetic Monitoring experienced degraded browser check performance due to a faulty release that has been rolled back.

This impacted all regions, specifically the probes. The API itself experienced no issues.

1762966243 - 1762966243 Resolved

Grafana k6 Degraded Performance

This incident has been resolved.

1762335156 - 1762353237 Resolved
⮜ Previous Next ⮞