Incident History

Metrics Write Outage in Multiple Cells

As of 23:07 UTC we are considering this incident as resolved.

Mitigation efforts have restored normal write performance, and error rates have returned to expected levels.

We have confirmed stability across the affected areas and continue to monitor, but no further impact is expected.

If you continue to experience any issues, please reach out to support.

1763409762 - 1763420920 Resolved

Hyderabad Probe Issues

We experienced degraded service with the Hyderabad probe today around 13:20 UTC which was resolved as of 17:30 UTC.

1763406104 - 1763406104 Resolved

Cortex - read/write path disruption

This incident has been resolved.

1763388393 - 1763450782 Resolved

PDC-Prod-eu-west-2 cluster degraded performance

This incident has been resolved.

1763380560 - 1763382181 Resolved

Loki Prod 012 read-path-unstable

resolved since 03:02UTC

1763352772 - 1763355349 Resolved

Degraded Brower Check Performance

Spanning from November 10th, 18:00 UTC to November 11th, 22:00 UTC, Synthetic Monitoring experienced degraded browser check performance due to a faulty release that has been rolled back.

This impacted all regions, specifically the probes. The API itself experienced no issues.

1762966243 - 1762966243 Resolved

Grafana k6 Degraded Performance

This incident has been resolved.

1762335156 - 1762353237 Resolved

Grafana K6 Tests Not Starting

Test runs are now working as expected. The duration of this incident was roughly 30 minutes. The test runs were not able to start, and the app and the API were not accessible.

1762282804 - 1762284438 Resolved

Synthetic Monitoring Checks North Virginia Probe Down In prod-us-east-0

The incident is resolved as of 14:35 UTC.

1762265267 - 1762277987 Resolved

Loki-Managed Rules Failed For Some Tenants

From Oct 31st 17:40 UTC to Nov 3rd 14:50 UTC:

Due to some internal auth issues, the components evaluating loki-managed rules failed to push the evaluated recording and alert rules to the metrics endpoint for some tenants.

1762182264 - 1762182264 Resolved
⮜ Previous Next ⮞