After a thorough investigation, we have determined that the recent connectivity issues experienced for statuspage.io domains in Brazil were isolated to specific Internet Service Providers (ISPs) within Brazil and there was no impact or issue with Statuspage’s infrastructure and DNS configurations itself. Our team is actively monitoring the situation and trying our best to reach out to the relevant ISPs to ensure that access issues are resolved for all affected users. If connectivity issues persist, and you are located in Brazil, we kindly ask that you open a support request with your internet service provider or follow our community post to resolve the issue.
There was an intermittent error with status embed page where cached responses were incorrectly being shown to a few customers. This is resolved as of now.
From 06:00 UTC to 07:45 UTC on October 28, 2023, Atlassian customers using Statuspage had intermittent issues with all Statuspage functionality. The event occurred due to a database performance issue during a scheduled database maintenance. This impacted customers in all regions. The incident was detected within one minute by monitoring the upgrade process and mitigated by rolling back to a known good snapshot which put Statuspage systems into a known good state. The total time to resolution was about one hour and 45 minutes.
IMPACT
The overall impact was between 06:00 UTC and 07:45 UTC October 28, 2023. This incident affected Statuspage customers from all regions and caused intermittent backend errors on all Statuspage activity including viewing pages, adding subscribers, and creating/updating events. We performed a rollback operation during recovery to return to a known good state.
ROOT CAUSE
The issue was caused by database performance issues after a routine database maintenance and upgrade. As a result, our backends returned intermittent errors to several user requests.
REMEDIAL ACTIONS PLAN & NEXT STEPS
We take the utmost care to provide a highly reliable service. We will pursue several preventive measures to ensure that this situation does not occur in the future, including:
Fixing the cause of the performance issues before future upgrades; and
Improving our testing process for database upgrades to catch potential performance issues.
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes.
IMPACT
The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases.
ROOT CAUSE
The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service.
REMEDIAL ACTIONS PLAN & NEXT STEPS
We are continuously improving our system's resiliency.
We are prioritizing the following improvement actions to avoid repeating this type of incident:
Audit and improve service rate limits and client retry and backoff behavior.
Improve scale and load test automation for complex service interactions.
Audit cross-service dependencies and minimize them where possible related to sign-in flows.
Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to:
Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available.
Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled.
We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability.