Fly machines API slow/unresponsive

Resolved

This incident is now resolved.

Update

After reverting the suspect changes, our metrics indicate service recovery. The Machines API is now operating normally. We are monitoring for any lingering issues.

Update

We've identified the problematic change and are performing a rollback to restore service.

Investigating

Fly API operations are currently slow or failing, we're investigating the cause.

This will affect most operations that rely in the machines API, including starting Fly builders, deploying, creating/destroying machines, querying machine or volume data via the API.

Existing, running apps and machines are not affected and continue to serve normally.