Delayed metric reporting in NRT and SIN regions

Resolved

This incident has been resolved. All hosts in SIN and NRT are reporting up to date metrics.

Update

Currently one host in SIN is still finishing working through it's metrics backlog and is reporting delayed metrics. Other hosts in NRT and SIN are reporting metrics correctly.

If needed, users with impacted machines on the remaining host can use fly machine clone to create new machines in the region, which should land on a different host.

Update

Most hosts in NRT and SIN have completed backfilling their metrics and are up to date in fly-metrics.net.

Four hosts are still working through the backlog; machines on those hosts are still reporting delayed metrics at this time.

Update

We are continuing to process the metrics backlog in NRT and SIN. Progress is being made, but due to the volume of metrics this may still take some time to fully complete.

At this time users with machines on impacted hosts will see metrics beginning to backfill into fly-metrics.net. However many will not be fully caught up yet.

This impacts metrics only, the underlying machines continue to work normally.

Update

A small number of hosts in the NRT (Tokyo) and SIN (Singapore) are reporting delayed metrics to the hosted Grafana charts at fly-metrics.net. Users with machines on impacted hosts will see delayed or spotty metrics in their Grafana charts.

Only metrics for these machines are impacted. The underlying machines continue to receive and serve traffic as usual, and all machine actions(stopping, starting, deploys etc.) continue to work normally.

We are processing the backlog of metrics on these hosts, but metrics will be delayed until this is complete.