Delays for web and email notification delivery

Resolved

On April 29th, 2025, between 8:40am UTC and 12:50pm UTC the notifications service was degraded and stopped delivering most web and email notifications as well as some mobile push notifications. This was due to a large and faulty schema migration that rendered a set of database primaries unhealthy, affecting the notification delivery pipelines, causing delays in the most of the web and email notification deliveries.We mitigated the incident by stopping the migration and promoting replicas to replace the unhealthy primaries.In order to prevent similar incidents in the future, we are addressing the underlying issues in the online schema tooling and improving the way we interact with the database to not be disruptive to production workloads.

Investigating

The notification delivery backlog has been processed and notifications are now being delivered as expected.

Investigating

New notification deliveries are occuring in a timely manner and we have processed a significant portion of the backlog. Users may still notice delayed delivery of some older notifications.

Investigating

Web and email notifications continue to be delivered successfully and the service is in a healthy state. We are processing the backlog of notification deliveries which are currently as much as 30-60 minutes delayed.

Investigating

We are starting to see signals of recovery with delayed web/email notifications now being dispatched.The team continue to monitor recovery and ensure return to normal service.

Investigating

We are seeing impact on both web and email notifications with most customers seeing delayed deliveries. The last incident updated regarding impact on email notifications was incorrect. Email notifications have been experiencing the same delays as web notifications for duration of incident.We have applied changes to our system and are monitoring to see if these restore normal service. Updates to follow.

Investigating

Web notifications are experiencing delivery delays for the majority of customers. We are working to mitigate impact and restore delivery times back within normal operating bounds.Email notifications remain unaffected and are delivering as normal.We will provide futher updates as we have more information.

Investigating

We are currently investigating this issue.