Increased API Errors

Resolved

This incident has been resolved.

Update

Our metrics show that machines API has mostly recovered. We're still working through a small number of degraded Managed Postgres clusters.

Update

We have identified and implemented a fix for the root cause of this incident and are monitoring the results. Machines API should be seeing recovery.

On the Managed Postgres side, New MPG creates should be working. A small number of clusters remain degraded, or are experiencing connectivity issues. We are continuing to restore these as quickly as possible.

Update

We are continuing to work on a fix for this issue. Users will still see elevated error rates creating new apps, machines, and MPG clusters, as well as other API operations. Users may also see newly created machines remaining in a created state for an extended period before starting.

Update

We are continuing to work on a fix for this issue.

Update

We are continuing to see elevated errors with creating new apps, new MPG clusters, and setting secrets. A small number of MPG clusters continue to see connectivity errors. We are continuing to work on these issues.

Update

We continue to see increased error rates creating new apps and MPG clusters, as well as with setting secrets. We are continuing to work to resolve these issues.

A small number of MPG clusters are still experiencing connectivity issues. We are working to restore these to a healthy state as quickly as possible.

Update

We have identified some continuing issues with creating new apps and secrets and are working to resolve them.

Update

Our monitoring indicates that machines API, flyctl and dashboard access should have recovered. Some Managed Postgres clusters may still be affected -- Rest assured that your cluster's data is intact while we are working to recover them to a working state.

Update

We have identified and implemented a fix for the API outage. Some Managed Postgres clusters may be affected as a secondary effect and we are currently working to restore them.

Investigating

We are investigating an increase in errors that affects numerous API endpoints including the Machines API, flyctl and dashboard.