LetsEncrypt API Outage causing Delayed Cluster Modifications

Resolved

This incident has been resolved.

Update

We continue to see cluster modifications get applied successfully after LetsEncrypt has restored service. We will monitor the system and provide another update before 12UTC.

Update

LetsEncrypt has restored service and we are seeing cluster modifications get applied.

Update

LetsEncrypt is continuing to experience elevated errors. Our system continues to use Google Trust Services when possible; however, backup restores are dependent on LetsEncrypt and are blocked. New serverless and flex instances are failing to provision. New dedicated clusters are unaffected by this outage.

Google Trust Services certificate issuance is not available in Atlas for Gov. In Atlas for Gov, dedicated cluster provisioning is blocked.

Update

LetsEncrypt is continuing to experience elevated errors. Our system continues to use Google Trust Services when possible; however, backup restores are dependent on LetsEncrypt and are blocked. New serverless and flex instances are failing to provision. New dedicated clusters are unaffected by this outage.

Update

LetsEncrypt is continuing to experience elevated errors. This is now causing new serverless and flex instances to fail to provision. New dedicated clusters are unaffected by this outage.

Update

We have updated our system to fallback to Google Trust Services. Any new clusters provisioned now will proceed normally. Please recreate your cluster or resubmit your cluster topology change to bypass the LetsEncrypt outage.

Investigating

Our LetsEncrypt CA provider is experiencing an API outage https://letsencrypt.status.io/pages/55957a99e800baa4470002da. Atlas customers can submit cluster modifications but will experience a delay in these changes getting applied. Existing clusters are unaffected by this outage.