Incident History

The incident has been resolved.

This incident has been resolved.

Service Disruption Caused by ElastiCache Automated Certificate Rotation

Summary

Based on our initial analysis with AWS, this incident was caused by an automated certificate update for the ElastiCache middleware.

Timeline

At 02:58 AM on October 29 (Beijing Time), AWS initiated an automated certificate update for our ElastiCache instances. During this process, the primary and replica nodes of the ElastiCache cluster experienced issues, preventing backend services from accessing the component.

• Time of Impact: 02:59 - 03:22，03:42 - 03:48 (Beijing Time)
• Scope of Impact: Cloud Printing and MakerWorld

Next Steps & Action Items

We have raised two critical issues with AWS Support:

• The automated update occurred outside of our designated maintenance window.
• The instability of the primary-replica nodes during the update process.

AWS Support has escalated these issues to their internal engineering team for a detailed root cause analysis. We will provide further updates as soon as we receive more information from AWS.

Updated on November 13, 2025

• The issue occurred when a scheduled TLS certificate renewal disconnected the replication connection between primary and replica nodes. Certificate renewal process identified this outdated certificate and terminated the connection, triggering a full resynchronization of several hundred gigabytes data.
• During the full resynchronization period (02:59 - 03:22), ElastiCache applied connection throttling to prioritize replication traffic. This throttling caused client connections to timeout which leads to our cloud services outage.
• After the replica finished loading the RDB data, the primary began sending accumulated replication data. Connection throttling was applied again to manage the replication traffic (03:42 - 03:48), causing another brief connection storm until the transfer completed.

Between 08:02 and 11:52 UTC+8, some users experienced intermittent issues accessing our cloud services.

After a joint investigation with our cloud provider, AWS, we have confirmed the root cause was network instability from the carrier, Cogent. Access requests routed through the Cogent network were subject to timeouts and packet loss.

Due to several recent incidents involving this provider, AWS has proactively rerouted traffic away from Cogent to alternative network paths. This action significantly mitigates the risk of similar disruptions in the future.

Cogent Network Status: https://ecogent.cogentco.com/network-status

Following a joint investigation with our cloud provider, AWS, we have confirmed the root cause was network instability from the carrier, Cogent. This instability caused access requests routed through the Cogent network to experience timeouts and packet loss.

This is the same issue detailed in our another incident report: https://status.bambulab.com/incidents/s6wds2nqyght

This incident has been resolved. It lasted about 10 minutes.

This incident has been resolved.

This incident has been resolved.

Due to a surge in traffic on makerworld.com at 17:55 UTC, users are experiencing slow loading times, and some pages are not displaying correctly. This issue lasted for approximately 20 minutes.

This incident has been resolved.