Incident with Issues

Resolved

On July 21st, 2025, between 07:00 UTC and 09:45 UTC the API, Codespaces, Copilot, Issues, Package Registry, Pull Requests and Webhook services were degraded and experienced dropped requests and increased latency. At the peak of this incident (a 2 minute period around 07:00 UTC) error rates peaked at 11% and went down shortly after. Average web request load times rose to 1 second during this same time frame. After this period, traffic gradually recovered but error rate and latency remained slightly elevated until the end of the incident.This incident was triggered by a kernel bug that caused a crash of some of our load balancers during a scheduled process after a kernel upgrade. In order to mitigate the incident, we halted the roll out of our upgrades, and rolled back the impacted instances. We are working to make sure the kernel version is fully removed from our fleet. As a precaution, we temporarily paused the scheduled process to prevent any unintended use in the affected kernel. We also tuned our alerting so we can more quickly detect and mitigate future instances where we experience a sudden drop in load-balancing capacity.

Investigating

API Requests and Codespaces are operating normally.

Investigating

Copilot is operating normally.

Investigating

Webhooks is operating normally.

Investigating

Mitigations have been applied and we are seeing recovery. We are continuing to closely monitor the situation to ensure complete recovery has been achieved.

Investigating

Issues is operating normally.

Investigating

Packages is operating normally.

Investigating

We are currently implementing mitigations for this issue.

Investigating

Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating

We continue to investigate reports of degraded performance and intermittent timeouts across GitHub.com.

Investigating

Pull Requests is operating normally.

Investigating

We're continuing to investigate reports of degraded performance and intermitiant timeouts across GitHub.com.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Packages is experiencing degraded performance. We are continuing to investigate.

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Codespaces is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of degraded performance for Issues and Webhooks