Incident History

Depot builders failing with internal error

This incident has been resolved.

1739229715 - 1739233798 Resolved

SSH failing for newly created machines

This incident has been resolved.

1739223561 - 1739228030 Resolved

Elevated network latency in FRA

Network functionality is fully restored in FRA.

1738766650 - 1738771870 Resolved

Capacity Constraints in IAD

This incident has been resolved

1737663820 - 1737669991 Resolved

Deploys using Depot Builders failing

This issue has been resolved, deploys using Depot Builders are succeeding as expected.

1737655541 - 1737657346 Resolved

API errors

This incident has been resolved.

1737048625 - 1737049726 Resolved

Machine creation errors in LHR

We observed several periods where Machine creations in LHR resulted in authentication errors from 11 Jan to 15 Jan 2025. Customers creating new Machines in the region may have seen failures with:

failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token : verify: context deadline exceeded

The disruptions were caused by degraded connectivity to our token creation service from three hosts.

We deployed a preventative fix for the network issues on 15 Jan 2025 at 12:58 UTC.

Timestamps of occurrences (UTC):

2025-01-11 03:32 to 2025-01-11 04:11 2025-01-11 17:07 to 2025-01-11 17:54 2025-01-14 11:36 to 2025-01-14 12:14 2025-01-15 07:46 to 2025-01-15 09:49

1736966634 - 1736966634 Resolved

Bluegreen healthchecks not passing

This incident has been resolved.

1736947729 - 1736968941 Resolved

Network issues in SJC region

This incident has been resolved.

1736565606 - 1736573078 Resolved

Transient networking issue in FRA

This incident has been resolved.

1734706559 - 1734708247 Resolved
⮜ Previous Next ⮞