Metrics display issue in Mumbai (BOM) region

Resolved

This incident has been resolved and all hosts in BOM are accurately reporting metrics.

Update

One host in BOM remains reporting delayed metrics as it continues to catch up. All other hosts in BOM are reporting metrics correctly.

If needed, users with impacted machines can use fly machine clone to create new machines in the region, which should land on a different host.

Update

Metrics have returned to normal for most hosts in BOM. Two hosts are still reporting delayed metrics, but are continuing to catch up.

Users with impacted machines can use fly machine clone to add new machines, which should land on a different host.

Update

Metrics have completed backfilling and are up to date on most hosts in the BOM region. Two hosts are still working through the backlog; machines on those two hosts are still reporting delayed metrics at this time.

Update

We are continuing to process the metrics backlog in BOM. Progress is being made, but due to the volume of metrics this may still take some time to fully complete.

At this time users with machines in BOM should see metrics from the past 12h beginning to backfill into fly-metrics.net. However most will not be fully caught up yet.

Update

We are continuing to work through the backlog of metrics in BOM. Metrics will remain unavailable for BOM machines until this is complete.

Update

Our metrics cluster is continuing to working through the backlog of metrics in BOM. Metrics for machines running in the BOM region will continue to be unavailable until this is complete.

Update

The cause of the issue has been identified and a fix is being implemented.

Metrics for machines in the BOM region remain unavailable in fly-metrics.net, however the machines themselves continue to run, start, and stop normally.

Investigating

We are currently investigating issues collecting machine metrics for machines running in the BOM (Mumbai, India) region. Machines in the region continue to run, start, and stop normally, however metrics for these machines are not displaying in fly-metrics.net.