Capacity isssues in iad and dfw

Resolved

This incident has been resolved.

Update

We're continuing to monitor after having added more capacity to our DFW and IAD regions.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Update

We have added additional capacity in DFW and IAD regions and are monitoring the impact.

New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times.

Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.

Update

We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.

Update

New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.

Update

We have identified the problem and are working on a fix.