Delays starting Depot Builders in IAD

Resolved

This incident has been resolved.

Update

A fix has been implemented and we are seeing improvements in builder performance, latency, and error rates. We are continuing to monitor for a full recovery.

Customers still seeing issues can trigger a fly-hosted builder based deploy with fly deploy --depot=false.

Update

We have identified an issue causing delays or failures when starting depoting builders located in the IAD region. Customers with builders in IAD may see delays or timeouts starting builds during fly deploy. We are working on a fix.

In the meantime customers can trigger a fly-hosted builder based deploy with fly deploy --depot=false. You can also change your builder region away from IAD via the settings tab of your Fly dashboard. We recommend DFW or ORD as alternate builder regions at this time.

Please note, your builder region can differ from the region your machines are in.