Read and write path outage in Hosted Logs ap-south-1 region cells.

Resolved

The incident was caused by multiple ingesters being unavailable at the same time due to moving ingester pods between nodes. It's a regular operation, but in this particular case the ingester took an unexpected long time to restart which coincided with another ingester eventually restarting at the same time, causing an issue.

Update

Cluster fully operational.

Update

We're currently monitoring health of the cluster since the outage was resolved and the issue was identified.

Investigating

We faced an issue with cells in ap-south-1 Hosted Logs region. Between 8:55 and 9:07 UTC this region faced the complete read and write paths outage. Since then it fully recovered and services are fully operational again. We're investigating the root cause right now.