Elevated Machines errors in ORD

Resolved

This incident has been resolved. Machine placement improvements are now deployed in all regions.

Update

The fix has been deployed to ORD and we are monitoring results. New Machines will be placed using stricter memory thresholds. Existing Machines on impacted hosts will be migrated to new hosts on start and update. We are also provisioning additional capacity in ORD.

Update

We are deploying a fix to enforce stricter memory thresholds for Machine placements. This will steer new and migrated workloads towards hosts with optimal capacity.

Investigating

We are investigating high memory utilization on some hosts in ORD. Customers may notice timeouts or errors in reserving resources when updating or creating new Machines in the region.

We are rebalancing workloads in ORD to relieve memory pressure on impacted hosts.