Incident with Webhooks

Resolved

On October 9th, 2025, between 14:35 UTC and 15:21 UTC, a network device in maintenance mode that was undergoing repairs was brought back into production before repairs were completed. Network traffic traversing this device experienced significant packet loss.Authenticated users of the github.com UI experienced increased latency during the first 5 minutes of the incident. API users experienced up to 7.3% error rates, after which it stabilized to about 0.05% until mitigated. Actions service experienced 24% of runs being delayed for an average of 13 minutes. Large File Storage (LFS) requests experienced minimally increased error rate, with 0.038% of requests erroring.To prevent similar issues, we are enhancing the validation process for device repairs of this category.

Investigating

All services have fully recovered.

Investigating

Actions has fully recovered but Notifications is still experiencing delays. We will continue to update as the system is fully restored to normal operation.

Investigating

Actions is operating normally.

Investigating

Pages is operating normally.

Investigating

Git Operations is operating normally.

Investigating

Actions and Notifications are still experiencing delays as we process the backlog. We will continue to update as the system is fully restored to normal operation.

Investigating

Pull Requests is operating normally.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

We are seeing full recovery in many of our systems, but delays are still expected for actions. We will continue to update as the system is fully restored to normal operation.

Investigating

Webhooks is operating normally.

Investigating

Webhooks is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is operating normally.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

API Requests is operating normally.

Investigating

We identified a faulty network component and have removed it from the infrastructure. Recovery has started and we expect full recovery shortly.

Investigating

Pull Requests is experiencing degraded availability. We are continuing to investigate.

Investigating

Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating

Actions is experiencing degraded availability. We are continuing to investigate.

Investigating

We are investigating widespread reports of delays and increased latency in various services. We will continue to keep users updated on progress toward mitigation.

Investigating

Issues is experiencing degraded availability. We are continuing to investigate.

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Pages is experiencing degraded performance. We are continuing to investigate.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of degraded availability for Webhooks