Cortex - read/write path disruption
This incident has been resolved.
Incident History
PDC-Prod-eu-west-2 cluster degraded performance
This incident has been resolved.
Loki Prod 012 read-path-unstable
resolved since 03:02UTC
Degraded Brower Check Performance
Spanning from November 10th, 18:00 UTC to November 11th, 22:00 UTC, Synthetic Monitoring experienced degraded browser check performance due to a faulty release that has been rolled back.
This impacted all regions, specifically the probes. The API itself experienced no issues.
Grafana k6 Degraded Performance
This incident has been resolved.
Grafana K6 Tests Not Starting
Test runs are now working as expected. The duration of this incident was roughly 30 minutes. The test runs were not able to start, and the app and the API were not accessible.
Synthetic Monitoring Checks North Virginia Probe Down In prod-us-east-0
The incident is resolved as of 14:35 UTC.
Loki-Managed Rules Failed For Some Tenants
From Oct 31st 17:40 UTC to Nov 3rd 14:50 UTC:
Due to some internal auth issues, the components evaluating loki-managed rules failed to push the evaluated recording and alert rules to the metrics endpoint for some tenants.
No services are reachable in prod-us-central-7
This incident has been resolved.
Temporary Trace Ingestion Errors
From approximately 16:30-8:15 UTC, a configuration change inadvertently removed a required headless service for hosted traces in one of our production regions. This caused elevated error rates and increased service-level objective (SLO) burn for the trace ingestion pathway. The underlying issue was a mismatch in internal configuration references following a prior migration. Re-enabling the headless service restored normal operation.