MPG Degraded clusters in AMS, IAD and SIN regions

Resolved

All MPG clusters are back to full, normal operations.

Update

All MPG clusters are reachable.

Update

We are still continuing cleanup on some clusters.

Update

All cluster primary and pgBouncer machines are now healthy and operating normally.

We are still continuing cleanup on some clusters with lagging or degraded replicas, but this should not impact writes or reads to clusters.

Update

We are continuing to work on restoring all clusters to full health.

Update

With the underlying incident stabilizing (https://status.flyio.net/incidents/3npj6935byt4) we are seeing improvements amongst impacted clusters. We continue to work on restoring all clusters to full health.

Update

A number of clusters in IAD, AMS, and SIN regions continue to see degraded replicas and PGBouncers at this time. A smaller number of clusters in these regions are also seeing disruption to their primaries. We continue to work on restoring full cluster health in all regions.

Update

A small number of MPG clusters in the AMS and IAD region are currently in degraded states due to downstream impact from this Machines API issue: https://status.flyio.net/incidents/3npj6935byt4

Most of the impacted clusters may see a degraded replica or PG Bouncer in their statuspage. A very small number may be unable to connect to their MPG primary node, the team is working to restore connectivity as the top priority. Users may also see delays registering new clusters in these regions at this time.