Grafana Cloud Hosted Logs increased write latency in eu-west-3 region.

Resolved

We consider the incident as resolved now. With regards to the cause - a slow physical partition of the backend database, which is used by the control plane of a critical component caused increased latency and occasional overloading with subsequent failing of the write path. Once writes switched to a different partition, the latency dropped and error rate went down.

Update

Elevated latency issue no longer happened since the last update, but we're still monitoring the situation. The write path is operational now for eu-west-3 region.

Update

We're still actively trying to mitigate the root cause. Symptoms are recurring from time to time.

Update

The issue has been identified and a fix is being implemented.

Investigating

Since 9:15 UTC, Hosted Logs tenants in eu-west-3 region are experiencing an elevated latency in the write path. Our team is looking to identify and resolve the issue.