Incident History

Temporary DNS resolution failure

A BGP configuration error caused our Anycast DNS to route to some nodes without the proper DNS infrastructure. The issue was temporary and was resolved as soon as we removed that node from BGP.

1787256472 - 1787256472 Resolved

Oauth/Macaroon Errors from flyctl

This incident has been resolved.

1787234048 - 1787235403 Resolved

MPG (v1) partially down in ORD

This incident has been resolved.

1787210738 - 1787212674 Resolved

6PN Networking issue in YYZ

6PN networking issues between some machines in YYZ during a rollout which was rolled back once we noticed errors. During this time some machines were unable to talk to internal resources like other DBs, other apps or MPG clusters.

1787102137 - 1787102137 Resolved

No capacity in ARN

The capacity issue in the ARN region has been resolved.

1786972775 - 1786981289 Resolved

Secrets service outage

This incident has been resolved.

1786735844 - 1786739625 Resolved

IPv6 Networking Issues

This incident has been resolved.

1786643109 - 1786650070 Resolved

Increased app-not-found errors

This incident has been resolved.

1786243805 - 1786259408 Resolved

MPG IAD data plane degraded for new clusters

Etcd is stable. Services are back to normal.

1785961561 - 1785962465 Resolved

MPG creation is failing in GRU due to lack of capacity

This incident has been resolved.

1785870221 - 1785900171 Resolved
⮜ Previous