Elevated GraphQL API Latency and Dashboard Error Rates

Resolved

This incident has been resolved. We have seen API latency normalize and remain normal since ~19:00 UTC.

Update

API error rates have normalized, however users may still see elevated latency reaching some GraphQL endpoints. Latency continues to trend in the right direction, we continue to monitor for full recovery .

Update

We have deployed an initial fix and are seeing improvements. GraphQL error rates and latency remain elevated over the baseline at this time. We are continuing to keep a close eye on recovery.

Update

The issue has been identified and a fix is being implemented.

Investigating

We are continuing to investigate elevated latency and error rates on our GraphQL API endpoints. Users may see errors on parts of the platform that use these APIs. This includes Flyctl actions such as deploys, as well as the fly.io dashboard.

Investigating

We are investigating elevated API Latency and Error rates on the platform. Users may see delays or errors creating apps, as well as on some dashboard pages.

Machines API actions appear unimpacted at this time