Disruption with some GitHub services

Resolved

On February 25th, 2025, between 14:25 UTC and 16:44 UTC email and web notifications experienced delivery delays. At the peak of the incident the delay resulted in ~10% of all notifications taking over 10 minutes to be delivered, with the remaining ~90% being delivered within 5-10 minutes. This was due to insufficient capacity in worker pools as a result of increased load during peak hours.We also encountered delivery delays for a small number of webhooks, with delays of up-to 2.5 minutes to be delivered.We mitigated the incident by scaling out the service to meet the demand.The increase in capacity gives us extra headroom, and we are working to improve our capacity planning to prevent issues like this occurring in the future.

Investigating

Web and email notifications are caught up, resolving the incident.

Investigating

We're continuing to investigate delayed web and email notifications.

Investigating

We're continuing to investigate delayed web and email notifications.

Investigating

We're investigating delays in web and email notifications impacting all customers.

Investigating

We are currently investigating this issue.