Delay in notification deliveries


Incident resolved in 3h15m46s

Resolved

On November 12th, 2025, from 13:10 - 17:40 UTC, notifications service was degraded, showing an increase in web notifications latency and increasing delays in notification deliveries. A change to the notifications settings access path introduced additional load to the settings system, degrading its response times. This impacted both requests to web notifications (with p99 response times as high as 1.5s, while lower percentiles remained stable) and notification deliveries, which reached a peak delay of 24 minutes on average. System capacity was increased around 15:10 UTC and the problematic change was fully reverted soon after that, restoring the latency of web notifications and increasing notification delivery throughput, decreasing the delay in notification deliveries. The notification queue was fully emptied around 17:40 UTC.We are working to adjust capacity in the affected systems and to improve the time needed to address these capacity issues.

1762969180

Investigating

We are continuing to monitor our mitigations to delays in notification deliveries. Some users may still experience delays of over 10 minutes.

1762967722

Investigating

We are continuing to work on mitigating delays in notification deliveries. Some users may still experience delays of over 10 minutes.

1762961706

Investigating

We are continuing to work on mitigating delays in notification deliveries. Some users may experience delays of over 10 minutes.

1762959531

Investigating

We are investigating delays of up to 10 minutes in notification deliveries. Our team has identified the likely cause and is actively working to mitigate the issue.

1762957514

Investigating

We are currently investigating this issue.

1762957434