Networking and metrics degradation

Resolved

This incident has been resolved.

Update

Network performance across Fly.io has returned to normal. This incident primarily impacted machines in the SJC and EWR regions.

Metrics are largely caught up, but users may still see a slight delay in reporting as the cluster finishes catching up.

Investigating

A change has been made and metrics on fly-metrics.net are backfilling. Users may still see a slight delay or gaps in new metrics being reported as the backfill completes.

We continue to see higher than usual latency and packetloss across the network. We are continuing to investigate.

Investigating

We are currently investigating increased latency and packet loss across multiple regions. Customers may see additional latency on requests at this time.

Relatedly, prometheus metrics reported via fly-metrics.net is currently degraded. Users may see delays or gaps in metrics at this time. We are working to address both issues.