Incident with Webhooks

Resolved

On January 9, 2025, between 01:26 UTC and 01:56 UTC GitHub experienced widespread disruption to many services, with users receiving 500 responses when trying to access various functionality. This was due to a deployment which introduced a query that saturated a primary database server. On average, the error rate was 6% and peaked at 6.85% of update requests.We mitigated the incident by identifying the source of the problematic query and rolling back the deployment.We are investigating methods to detect problematic queries prior to deployment to prevent, and to reduce our time to detection and mitigation of issues like this one in the future.

Investigating

We have identified the root cause and have deployed a fix. Majority of the services have recovered. Actions service is in the process of being recovered.

Investigating

Copilot is operating normally.

Investigating

Issues is operating normally.

Investigating

Pages is operating normally.

Investigating

Git Operations is operating normally.

Investigating

Webhooks is operating normally.

Investigating

Pull Requests is operating normally.

Investigating

Codespaces is operating normally.

Investigating

We have identified the root cause and have deployed a fix. Service are recovering.

Investigating

API Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

We are continuing the investigation of multiple service issues. We will continue to keep users updated on progress towards mitigation.

Investigating

Copilot is experiencing degraded performance. We are continuing to investigate.

Investigating

Codespaces is experiencing degraded performance. We are continuing to investigate.

Investigating

Codespaces is experiencing degraded availability. We are continuing to investigate.

Investigating

Git Operations is experiencing degraded availability. We are continuing to investigate.

Investigating

We are investigating reports of issues with multiple services including authentication, PRs, codespaces, pages, git operation, and apis. We will continue to keep users updated on progress towards mitigation.

Investigating

Pages is experiencing degraded performance. We are continuing to investigate.

Investigating

Git Operations is experiencing degraded performance. We are continuing to investigate.

Investigating

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Investigating

Issues is experiencing degraded performance. We are continuing to investigate.

Investigating

Actions is experiencing degraded performance. We are continuing to investigate.

Investigating

We are investigating reports of degraded availability for Webhooks