Our Engineering team identified and resolved an issue impacting multiple services, in multiple regions.
From 19:53 - 20:22 UTC, users may have experienced errors with the following operations, for all regions:
Droplet creates with a root password
Droplet root password resets
Droplet Snapshots
MongoDB cluster creates
Let’sEncrypt certificate creation (for Load Balancers and Spaces)
LoadBalancer creation with certificates
Spaces bucket and key creations
Fetching CSV invoices
Bulk actions in the Spaces UI
Additionally, users could have seen latency while creating and updating Apps.
Our Engineering team quickly identified the root cause of the incident to be an incorrect firewall policy applied to core infrastructure and rolled back the change.
We apologize for the disruption. If you continue to experience any issues, please open a Support ticket from within your account.
Our Engineering team has confirmed that this incident has been fully resolved.
We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.
Our Engineering team has confirmed the full resolution of the issue impacting the Managed Kubernetes cluster creation. As of 12:18 UTC the functionality has been restored completely and users should be able to deploy new clusters via Cloud Control Panel and API.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
As of 13.10 UTC, our Engineering team has confirmed full resolution of networking in our SFO region.
If you continue to experience problems, please open a ticket with our support team from your Cloud Control Panel. Thank you for your patience and we apologize for any inconvenience.
From 17:02 - 18:44 UTC, the Spaces API experienced availability dips. These dips caused Spaces bucket operations in SGP1 for a subset of users to experience latency or 503 errors.
The fix implemented by our Engineering team has been monitored and availability for the Spaces API is stable. Users should be able to interact with their Spaces buckets successfully.
If you continue to experience any issues, please reach out to the Support team from within your account.
Our Engineering team has confirmed that this incident has been fully resolved.
We appreciate your patience throughout this process and if you continue to experience problems, please open a ticket with our support team for further review.
Our Engineering team identified and resolved an issue impacting multiple pages in the Cloud Control Panel.
From 16:12 - 16:26 UTC, users may have experienced access denied errors while loading pages associated with different services on the Cloud Control Panel. The impacted pages included Managed Database, Block Storage Volumes, Reserved IPs, Domains, Settings, App Platform, Applications & API, Container Registry, Functions, Managed Kubernetes and Monitoring.
Our Engineering team was able to take quick action to mitigate the impact and resolve the issue.
Thank you for your patience, and we apologize for any inconvenience. If you continue to experience any issues, please open a Support ticket for further analysis.
From 13:45 - 16:04 UTC, users were unable to sign up for new DigitalOcean accounts at https://cloud.digitalocean.com and existing users were unable to update credit card information within the Cloud Control Panel, due to an issue with a billing verification key. This issue is now fully resolved and users should be able to interact with the Cloud Control Panel normally.
We apologize for the inconvenience and invite users to retry any failed attempts. If you continue to experience issues or have any questions, please reach out to Support from within your account.