Fly Metrics' ingestion is stopped working

Resolved

This incident has been resolved.

Update

We are processing the backlog and metrics are mostly backfilled.

Update

Metrics writes have been queued and we are still processing them. We are going to scale up the cluster to catch up the backlog.

Update

The issue is in our Victoria Metrics cluster's storage nodes. We've upgraded and rebooted the nodes.

Investigating

Fly Metrics's ingestion is stopped working. While you can access https://fly-metrics.net/, you may not able to see new data points since around 06:50 PM UTC. We are actively working on the issue.