Macaroon Auth + Machines API Issues

Resolved

This incident has been resolved and we are seeing all platform functions operate normally.

Update

A fix has been implemented and we are monitoring the results.

Update

We have deployed another change and are seeing wider improvements in platform stability across all regions. Performance is trending to normal, though users may still see some degradation at this time.

We are continuing to closely monitor to ensure full, stable recovery. We will provide another update in 15m.

Update

We are seeing elevated cluster errors with Managed Postgres clusters as the MPG control plane recovers from the API outage. MPG Users may see elevated rates of failing or slow connections, as well as increased primary/replica failovers. The managed postgres team is addressing any degraded clusters.

We will provide a further update within 15m.

Update

We continue to seeing degraded performance and increased errors with the Machines API and other platform features at this time. We are continuing to work on fully restoring service.

Update

An initial fix has been deployed and we are starting to see platform features recover. Users may still see degraded performance and intermittent failures at this time.

We are continuing to address the issue to ensure a full stable recovery.

Update

We have identified the cause of the issue and are working on deploying a fix. Impacted features remain unavailable or degraded at this time.

Already running customer applications/machines remain available. MPG clusters remain generally reachable and healthy, however new clusters cannot be provisioned and failovers may not complete.

We will provide another update within 15 minutes.

Update

We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing.

Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running.

We will provide another update within 15 minutes

Investigating

We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.