Low capacity in IAD

Resolved

This incident has been resolved.

Update

With the additional capacity we've brought online, machine start failure rates in IAD have now recovered. We'll continue to monitor IAD capacity.

Update

We've brought some additional capacity online in IAD and are seeing improvements, and we're continuing to work on adding more and freeing up additional room.

Investigating

We're continuing to evaluate our options for increasing short-term capacity in the IAD region.

Investigating

We're currently investigating capacity issues in IAD that is preventing machine starts (machine creates are currently unaffected). This may result in deploys failing to complete (even for apps outside of the IAD region). As a workaround, using legacy Fly builders explicitly located in another region (i.e., FLY_REMOTE_BUILDER_REGION=lhr fly deploy --depot=false --recreate-builder) may help in the meantime.