Deploys hanging at waiting for Depot Builder

Resolved

This incident has been resolved.

Update

The fix has been rolled out and we are seeing deploys using depot builder succeeding normally. We continue to monitor to ensure full recovery.

Depot builders have been reenabled as the default option for new deploys

Update

A fix is being rolled out. Fly builders continue to be the default while this is deployed

Update

We are again seeing elevated latency provisioning depot builders on new deploys. Users may see deploys using Depot builders hang or timeout at the "Waiting for Depot Builder" step. We are working on a fix.

We are switching all deploys to use the default Fly builders in the meantime.

If desired users can manually switch back to depot builders using fly deploy --depot=true but may continue to see latency issues at this time.

Update

We have seen elevated latency provisioning Depot builders during deployments over the past hour. This caused some deploys to hang or timeout at the "Waiting for Depot Builder" step in this period.

Latency has improved and builder provision times are back to normal. We're continuing to monitor to ensure latency remains normal.