Our Engineering team has confirmed that the issues with Spaces CDN functionality has been fully resolved. Users should now be able to use CDN functionality normally.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Our Engineering team has confirmed that the issues with Authoritative DNS resolution in NYC1 has been fully resolved. DNS queries should now be resolving normally.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Our Engineering team has confirmed the full resolution of the issue impacting Managed Database Operations, and all systems are now operating normally. Users may safely resume operations, including upgrades, resizes, forking, and ad-hoc maintenance patches.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Our Engineering team has resolved the issue impacting prohibiting a subset of users from logging in, and the login flow is now operating normally.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
From 16:10 UTC to 22:46 UTC, users may have experienced issues while executing Managed Database CRUD Operations.
Our Engineering team has confirmed the full resolution of the issue impacting Managed Database CRUD Operations, and all systems are now operating normally. Users may safely resume operations, including upgrades, resizes, forking, and ad-hoc maintenance patches.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Our Engineering team identified an issue affecting App Platform deployments across all regions. The deploy-on-push functionality may not have triggered new builds between 15:45 and 18:15 UTC.
A fix has been implemented, and as of 18:15 UTC, deployments have been functioning successfully. If you continue to experience problems, please open a ticket with our support team. . We apologize for any inconvenience.
Our Engineering has confirmed full resolution of the issue. Users should now be able to manage their Reserved IPs via the Cloud Panel without issue.
If you continue to experience any issues, please open a support ticket from within your account. Thank you.
On Thursday, November 28, 2024, our domain registrar, Network Solutions, made an update to our digitaloceanspaces.com domain. The change made at 22:38 UTC (5:38:40PM ET) added an Extensible Provisioning Protocol (EPP) clientHold status code to our domain.
The code prevents DNS resolution until a customer contacts Network Solutions to clear the hold. The impact of the hold resulted in DigitalOcean customers being unable to sign-up/log-in via the Cloud Control Panel at the beginning of the incident and experiencing errors with Spaces buckets and dependent services, such as DigitalOcean Container Registry, App Platform, Functions, Load Balancers, and Mongo Managed Database Backups for the duration of the incident.
Root Cause: Domain registrar erroneously placed a clientHold on the digitaloceanspaces.com domain, impacting DNS resolution and causing an outage.
Impact: Traffic to DigitalOcean products (e.g. Cloud Control Panel, Spaces, App Platform, etc.) began to see intermittent failures between 11/28 22:38 UTC and 04:03 UTC, with the worst failures happening between 03:00 and 04:03 UTC as DNS cache and TTLs began expiring.
Response: DigitalOcean escalated to the domain registrar, as well as Verisign (as the authoritative domain registry for .com domains), to have the clientHold removed.
November 28, 2024 (UTC)
November 29, 2024 (UTC)
DigitalOcean teams are working on multiple types of remediation to prevent a similar incident from happening.
DigitalOcean is working with Network Solutions to understand what happened on their end that resulted in the clientHold being applied to our domain incorrectly.
In addition, we are reviewing other domain registrars as possible new homes for our domains.
Teams are also reviewing our monitoring and alerting to reduce our time to detect incidents related to registrar imposed changes and/or DNS resolution.
Our Engineering team identified and resolved the networking issue in our SFO3 region.
From 10.02 UTC to 11:44 UTC, users may have experienced connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.
The impact has been mitigated and services should be working normally at this time.
If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.
Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k
From 20:23 - 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.
The impact has been mitigated and services should be working normally at this time.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.