Hyderabad Probe Issues
We experienced degraded service with the Hyderabad probe today around 13:20 UTC which was resolved as of 17:30 UTC.
Incident History
Cortex - read/write path disruption
This incident has been resolved.
PDC-Prod-eu-west-2 cluster degraded performance
This incident has been resolved.
Loki Prod 012 read-path-unstable
resolved since 03:02UTC
Degraded Brower Check Performance
Spanning from November 10th, 18:00 UTC to November 11th, 22:00 UTC, Synthetic Monitoring experienced degraded browser check performance due to a faulty release that has been rolled back.
This impacted all regions, specifically the probes. The API itself experienced no issues.
Grafana k6 Degraded Performance
This incident has been resolved.
Grafana K6 Tests Not Starting
Test runs are now working as expected. The duration of this incident was roughly 30 minutes. The test runs were not able to start, and the app and the API were not accessible.
Synthetic Monitoring Checks North Virginia Probe Down In prod-us-east-0
The incident is resolved as of 14:35 UTC.
Loki-Managed Rules Failed For Some Tenants
From Oct 31st 17:40 UTC to Nov 3rd 14:50 UTC:
Due to some internal auth issues, the components evaluating loki-managed rules failed to push the evaluated recording and alert rules to the metrics endpoint for some tenants.
No services are reachable in prod-us-central-7
This incident has been resolved.